HOW-TO Make a photometric source catalog

The very first processing step in the photometric pipeline always consists of deriving catalogs from standard field observations. These observations should be fully reduced, i.e. de-biased and flatfielded (ReducedScienceFrame objects), and an astrometric calibration should also be available for the observations (AstrometricParameters objects). This step in the processing is the only time when image data actually enters the photometric pipeline. After this point, the rest of the photometric pipeline only works with the catalogs produced in this very first step.

Content of the Photometric Source Catalog

The Photometric Source Catalog contains one row per identified standard star with 8 values. These 8 columns are described in Table 1.

Table 1: Description of the 8 columns in a Photometric Source Catalog.

Column name Description
index index number of the star as listed in the Photometric Reference Catalog
origin origin of stellar magnitude as listed in Photometric Reference Catalog:
  Landolt: Landolt catalog,
  Stetson: Stetson catalog,
  AW2S: Preliminary Catalog for OmegaCAM Secondary Standards
ra Right Ascension (deg)
dec Declination (deg)
mag stellar magnitude in Photometric Reference Catalog,
  if a transformation table has been applied the value of mag incorporates this transformation.
mag_err error on mag
instmag -2.5*log10(countrate)
instmag_err error on instmag

Making photometric catalogs from the awe-prompt

The catalogs produced in this first processing step are represented by the PhotSrcCatalog class. This class is the real ‘working’ class in the photometric system because instances of it are used throughout. These photometric catalogs are the result of associating a SExtractor catalog with a photometric standard star catalog. The following sub-sections describe the two ways in which such catalogs can be produced from the awe-prompt.

Using a pre-cooked recipe

Deriving a photometric catalog from the awe-prompt using a pre-cooked recipe is done thus:

1. awe> from import PhotcatTask
2. awe> task = PhotcatTask(instrument = 'WFC',
   ...                     raw_filenames = ['r336603_3.fits'])
3. awe> task.execute()

To get detailed information about the use of the task, type:

awe> help(PhotcatTask)

which will, for example, show the parameters that can be passed to the constructor of the task.

Using the basic building blocks

A more elaborate but also very enlightening way of making a photometric catalog is by using the basic building blocks of the photometric pipeline themselves. This allows tweaking of the processing down to the nitty-gritty details. The most common way of deriving a photometric catalog in this way is given here:

 1. awe> from astro.main.PhotSrcCatalog import PhotSrcCatalog
 2. awe> from astro.main.PhotRefCatalog import PhotRefCatalog
 3. awe> from astro.main.AstrometricParameters import AstrometricParameters
 4. awe> from astro.main.ReducedScienceFrame import ReducedScienceFrame
 5. awe> frame = (ReducedScienceFrame.filename == 'r336603_3.reduced.fits')[0]
 6. awe> query = (AstrometricParameters.reduced == frame) &
    ...          (AstrometricParameters.is_valid == 1)
 7. awe> astrom_params = query.max('creation_date')
 8. awe> refcat = PhotRefCatalog.get()
 9. awe> photcat = PhotSrcCatalog()
10. awe> photcat.refcat = refcat
11. awe> photcat.frame = frame
12. awe> photcat.astrom_params = astrom_params
13. awe> photcat.refcat.retrieve()
14. awe> photcat.frame.retrieve()
15. awe> photcat.frame.weight.retrieve()
16. awe> photcat.make()
17. awe> photcat.commit()

In lines (1)-(4) the relevant classes are imported, and in steps (5)-(8) the necessary dependencies are retrieved from the database. In lines (9)-(16), a PhotSrcCatalog object is instantiated, its dependencies are set, and the make method is called. In step (17), the PhotSrcCatalog object is committed to the database. Note the explicit retrieve calls in steps (13)-(15) that access the fileserver.

It is important to realize that the separate steps detailed here are roughly the same as the ones performed by the PhotcatTask.

Configuring the photometric catalog

Any PhotSrcCatalog object has two knobs that allow the user to configure the behaviour of the make method and its results. The first one of these deals with configuring SExtractor, the other one with configuring the make method itself.

SExtractor can simply be configured through the sexconf attribute of the PhotSrcCatalog object. To give an example of how this works:

awe> photcat.sexconf.PHOT_APERTURES = 30

which tells SExtractor to use an aperture with a diameter of 30 pixels in measuring the FLUX_APER of the standard stars. More information about the configuraton of SExtractor can be found in the SExtractor manual.

The configuration of the make method itself is done through the process_params attribute of the PhotSrcCatalog object. To get information about the available configuration options, just type:


which will show the available configurable parameters, their meaning, and their default setting.

Inspecting the contents of the photometric catalog

To view the content of a PhotSrcCatalog object after it has been made, simply invoke its inspect method:

awe> photcat.inspect()

which will result in an output to screen that looks like the one shown in Figure 1. The inspect plot shows the magnitudes of the individual standard stars as known to the standard star catalog on the x-axis, and their associated raw zeropoints on the y-axis. The inspect method of course also functions on PhotSrcCatalog objects retrieved from the database.

the *inspect* method of a PhotSrcCatalog object

Figure 1: The result of invoking the inspect method of a PhotSrcCatalog object.

More visualisation options

The inspect method of a PhotSrcCatalog object re-directs the actual work to a dedicated plot object. This plot object is, by default, configured to show its output on screen with the individual raw zeropoints and magnitudes of the stars contained in the catalog on the vertical and horizontal axis, respectively. However, more visualisation options are available if one uses the plot object directly.

A plot object for PhotSrcCatalog can be retrieved through a factory function:

awe> from astro.plot.PhotometryPlot import create_plot
awe> plot = create_plot('Photcat')

whereafter it will be ready for use. The method to be called for plotting the contents of the PhotSrcCatalog object is:


where photcat is a valid PhotSrcCatalog object. Note that calling the show method just like that will just produce the same result as the inspect method of the PhotSrcCatalog object.

Using the plot object allows the user to plot several other quantities on the horizontal axis besides the magnitudes of the standard stars. Other quantities that can be plotted are the X-position, Y-position, and radial position of the stars on the chip. The plot object can be configured to do that as follows:

awe> plot.plot_params.XAXIS_TYPE = 'XPOS'

with MAGS, XPOS, YPOS and RADIAL as possible values. The default used by the inspect method is always MAGS.

The plot object can also be configured to send its output to a postscript file with a user-specified name:

awe> plot.plot_params.FILE_OUTPUT = 1
awe> plot.plot_params.FILE_NAME = 'my_photcat.eps'

where the FILE_OUTPUT toggle is set to True.

Query methods

Any given PhotSrcCatalog object features a large collection of methods that allow access to its data contents (many of these are actively used by the photometric pipeline). The most important simple methods are:

  1. photcat.get_number_of_sources(), which returns the number of sources in he catalog.
  2. photcat.get_source_attributes(), which returns information of the data content of a source. The information is stored in a dictionary with the attribute names of the source as keys and their types as values. To just get the attribute names, do: photcat.get_source_attributes().keys().
  3. photcat.get_dict_of_raw_zeropoints(), which returns a dictionary with the raw zeropoints of the sources in the catalog as values, and their indices as keys.
  4. photcat.get_median_raw_zeropoint(), which returns the median raw zeropoint of the sources within the catalog.
  5. photcat.get_average_raw_zeropoint(), which returns the average and standard deviation of the raw zeropoint distribution.
  6. photcat.make_skycat(), which dumps the catalog for overplotting in skycat.

The most flexible way of retrieving source information from the catalog is through the following query method:

awe> photcat.get_source_data(column_list)

which will return a dictionary with the indices of the sources as keys, and as values lists of the requested data items in the same order as specified in the input column_list. The input column_list is the list of data items to be retrieved. The entries in column_list should match the keys of the dictionary that is generated by the get_source_attributes method. These entries are strings.

Querying the database

Just like any other persistent data item in the system, a PhotSrcCatalog object has a large collection of attributes that can be used to retrieve it from the database. The most important of these are the instrument, filter, chip and date_obs attributes. The value of the latter corresponds exactly to the DATE_OBS attribute of the ReducedSciencFrame object that went into making the catalog, and serves as the master timestamp for PhotSrcCatalog objects.

Example queries

In the first example, the PhotSrcCatalog object derived from a science frame taken at 2003-02-11T21:00:00 is retrieved from the database. Only the one that has been marked as valid is requested.

awe> date_obs = datetime.datetime(2003,2,11,21)
awe> photcat = (PhotSrcCatalog.date_obs == date_obs) &\
...            (PhotSrcCatalog.is_valid == 1)

In the second example, the PhotSrcCatalog objects for the night from 2003-02-11 to 2003-02-12 are retrieved, and from these only those that are derived from science frames observed through the WFC broad-band Sloan G filter and from chip A5382-1-7.

awe> date_start = datetime.datetime(2003,2,11,12)
awe> date_end = date_start + datetime.timedelta(1)
awe> photcat = (PhotSrcCatalog.date_obs >= date_start) &\
...            (PhotSrcCatalog.date_obs <  date_end) &\
...            ( == '220') &\
...            ( == 'A5382-1-7') &\
...            (PhotSrcCatalog.is_valid == 1)

The last example shows the situation in which several PhotSrcCatalog objects have been made from one and the same science frame, and that only the last one created is wanted. The science frame has the name r300100_4.calib.fits.

awe> photcats = (PhotSrcCatalog.frame.filename == 'r300100_4.calib.fits')
awe> photcat = photcats.max('creation_date')