HOW-TO Create a ReducedScienceFrame

When an instrument is added to the Astro-WISE database environment, it is initialised (boot-strapped) with calibration frames that are valid “forever”. This allows one to always be able to reduce RawScienceFrames albeit with less than optimal results.

NOTE: In many cases, however, calibration files taken near to the date-obs of the science data can exist and be valid for that data. This may occur if another observer’s data has been reduced first. Although these calibration frames may give optimal results, it is always possible to create new calibration frames specific to the data being processed. The recipes will always choose the most recent calibration frames by default.

Making ReducedScienceFrames using the DPU

The most ideal way to process data in AWE is to process it with the DPU (see HOW-TO Process data in a distributed (parallel) way).

The DPU interface is built into AWE so that its use is simple and transparent. A basic DPU usage looks like:

awe> dpu.run('task_name', option1='', option2='', . . .)

For the ReduceTask, the DPU command would look more like:

awe> dpu.run('Reduce', d='2001-01-22', i='WFI', f='#841', o='AXAF')

or

awe> dpu.run('Reduce', i='WFI', raw_filenames=['WFI.2000-01-01T08:57:15.410_3.fits'])

The ReduceTask options via the DPU are as follows:

  • i (instrument, mandatory): string
  • d (date): string of the form CCYY-MM-DD
  • f (filter): string
  • raw_filenames (raw_filenames): list of strings
  • o (object): string with possible wildcards * and ?
  • oc (overscan): integer=0..10 (optional, default=6)
  • C (commit): integer=0..1 (optional, default=0)

The first example shows how to run the DPU ReduceTask using date, filter, and object information. This allows one to process all data for a given object on a specific day taken in a certain filter, for all CCDs of the mosaic. The second example shows how to run the DPU ReduceTask using specific data with known raw filenames.

The options can be used in any order and can be omitted (except i), but the likelihood of locating required data depends on relaying a minimum of information as shown in the examples.

Making a ReducedScienceFrame using the ReduceTask

Although the ideal method to process RawScienceFrames is the DPU, this task can be performed on a per chip basis. The ReduceTask is the main part of the recipe $AWEPIPE/astro/recipes/Reduce.py. Two examples of the syntax for the ReduceTask are given below:

awe> from astro.recipes.Reduce import ReduceTask
awe> task = ReduceTask(date='2001-01-22', chip='ccd51', filter='#841',
...                    object='AXAF', i='WFI')
awe> task.execute()

or

awe> task = ReduceTask(raw_filenames=['WFI.2000-01-01T08:57:15.410_3.fits'])
awe> task.execute()

The ReduceTask options are as follows:

  • date: string of the form CCYY-MM-DD
  • chip: string
  • filter: string
  • raw_filenames: list of strings
  • object: string with possible wildcards * and ?
  • overscan: integer=0..10 (optional, default=6)
  • commit: integer=0..1 (optional, default=0)

The first example shows how to run the ReduceTask using date, chip, filter, and object information. This allows one to process all data for a given object on a specific day taken in a certain filter, and only one CCD of the mosaic. The second example shows how to run the ReduceTask using specific data with known raw filenames.

The options can be used in any order and can be omitted, but the likelihood of locating required data depends on relaying a minimum of information as shown in the examples.

Making a ReducedScienceFrame using the basic building blocks

The third, and most powerful, way of creating a ReducedScienceFrame is by using the basic building blocks of the system directly from the awe-prompt. This method is used if one wants to manipulate the system down to the nitty-gritty details. An example of how a ReducedScienceFrame is made from the awe-prompt in this fashion is given below:

 1. awe> from astro.main.RawFrame import RawScienceFrame
 2. awe> from astro.main.ReducedScienceFrame import ReducedScienceFrame
 3. awe> from astro.main.MasterFlatFrame import MasterFlatFrame
 4. awe> from astro.main.ColdPixelMap import ColdPixelMap
 5. awe> from astro.main.HotPixelMap import HotPixelMap
 6. awe> from astro.main.BiasFrame import BiasFrame

 7. awe> raw  = (RawScienceFrame.filename == 'WFI.2000-01-01T08:57:15.410_3.fi
    ...  ts')[0]
 8. awe> hot  = HotPixelMap.select_for_raw(raw)
 9. awe> cold = ColdPixelMap.select_for_raw(raw)
10. awe> flat = MasterFlatFrame.select_for_raw(raw)
11. awe> bias = BiasFrame.select_for_raw(raw)

12. awe> reduced = ReducedScienceFrame()
14. awe> reduced.raw  = raw
15. awe> reduced.hot  = hot
16. awe> reduced.cold = cold
17. awe> reduced.bias = bias
18. awe> reduced.flat = flat
19. awe> reduced.raw.retrieve()
20. awe> reduced.hot.retrieve()
21. awe> reduced.cold.retrieve()
22. awe> reduced.bias.retrieve()
23. awe> reduced.flat.retrieve()
24. awe> reduced.set_filename()

25. awe> reduced.make()
26. awe> reduced.store()
27. awe> reduced.weight.store()
28. awe> reduced.commit()

In steps (1)-(6), the minimum number of classes relevant for this processing step are imported. Other classes that could be used here are FringeFrame and IlluminationCorrectionFrame. In steps (7)-(11), the database is queried for the necessary dependencies. In step (7), the RawScienceFrame to be processed is retrieved, and in steps (8)-(11) the calibration files to use. In steps (12) and (13), the ReducedScienceFrame object is instantiated, and one of its process configuration parameters is set. In steps (14)-(24), the dependencies of the ReducedScienceFrame are set, and the pixel-data underlying the calibration files are retrieved from the fileserver. In step (24), the filename of the ReducedScienceFrame to be is created and set. Finally, in steps (25)-(28), the ReducedScienceFrame is made, its pixel-data is stored, and the object itself is committed to the database. Note that in step (27), the WeightFrame associated with the ReducedScienceFrame object is also stored; the weights are a very important by-product of making the ReducedScienceFrame that should never be forgotten.

Output Logs

In both the DPU and non-DPU processing methods, log files are created (of the form CCYYMMDD_hhmmsssss.log). For both cases, logs are printed to both the screen and the log file. For the non-DPU method, nothing needs to be done to retrieve the logs, they just come as they are created. For the DPUmethod, the processing is done remotely, and they have to be retrieved manually. There is a simple mechanism with which to do this, a method of the DPU object called get_logs(). When this method is invoked, all the finished jobs are retrieved one at a time [1], displayed to screen, and printed to the log file. An example of this mechanism is given below:

awe> dpu.jobids()
[313]
awe> dpu.get_logs()

<log printed to file and screen>

awe> dpu.jobids()
[]

For more complete information on the functionality of the DPU, see HOW-TO Process data in a distributed (parallel) way.

Viewing the results

Query for and retrieve ReducedScienceFrames

When one of the above commands are run, it should generally be run first without the commit switch set (C=0, the default) so as to test whether all steps were successful. Once the logs have been examined and no serious problems found, the command can be run with the commit switch set to C=1.

If all went well, the RawScienceFrame has had all the calibration frames available applied to it to create the ReducedScienceFrame. Now this frame can be retrieved from the data server for inspection. This is done by first querying the database for the ReducedScienceFrames just created:

awe> q = ReducedScienceFrame.raw.filename == 'WFI.2000-01-01T08:57:15.410_3.fits'
awe> len(q)
1
awe> q = ReducedScienceFrame.raw.filename.like('WFI.2000-01-01T08:57:15.410*')
awe> len(q)
8

Once the desired ReducedScienceFrame is in the query, it is a simple matter to look at the images statistics and retrieve it:

awe> for f in q:
...      print(f.imstat.mean, f.imstat.median, f.imstat.stdev)
111.557922989 111.463745117 8.56794497829
107.94240659 107.812011719 8.38193208213
106.126105882 106.035461426 7.48044620812
97.2398969938 97.0784301758 8.0901417073
98.4600191901 98.3441619873 7.53172611373
91.9374131475 91.8213806152 7.21070922762
118.980466828 118.880493164 8.97856755639
106.038520679 105.926620483 8.20778339642
awe> for f in q:
...      f.retrieve()
15:37:34 - Retrieving Sci-USER-WFI-------#877-ccd54-----Sci-53664.4795766.fits
15:37:37 - Retrieving Sci-USER-WFI-------#877-ccd55-----Sci-53664.4796525.fits
15:37:40 - Retrieving Sci-USER-WFI-------#877-ccd52-----Sci-53664.4797027.fits
15:37:43 - Retrieving Sci-USER-WFI-------#877-ccd53-----Sci-53664.4797485.fits
15:37:46 - Retrieving Sci-USER-WFI-------#877-ccd57-----Sci-53664.4801416.fits
15:37:49 - Retrieving Sci-USER-WFI-------#877-ccd56-----Sci-53664.4801463.fits
15:37:52 - Retrieving Sci-USER-WFI-------#877-ccd50-----Sci-53664.4803221.fits
15:37:55 - Retrieving Sci-USER-WFI-------#877-ccd51-----Sci-53664.4803202.fits

Display ReducedScienceFrames

Once the images are retrieved, they can be viewed in a number of ways. There currently exists no efficient built-in mechanism to view images in AWE, so external viewers viewers should be used. One external viewer in common use is ESO’s SkyCat tool. The syntax used to view one of the images (the first in the query in this case) with SkyCat is:

awe> os.system('skycat %s' % (q[0].filename))

To view the entire mosaic with SkyCat, the mosaic must first be converted into a multi-extension FITS (MEF) file. The entire process is illustrated below:

awe> newname = 'Sci-USER-WFI-#877-53664.5.fits'
awe> mef = Image(newname)
awe> mef.frames = q
awe> mef.make()
awe> os.system('skycat %s' % (newname))

Once SkyCat appears, click the “Display as one Image” button to see the entire image.

See HOW-TO View processing results as a mosaic (MEF file) for more detailed information.

Ask ReducedScienceFrames about themselves

The database objects that are created (ReducedScienceFrames in this example) hold information about their history. This information can be found by inspecting various attributes of the object. For example:

awe> q = (ReducedScienceFrame.filename != '') &
...      (ReducedScienceFrame.instrument.name == 'WFI')
awe> len(q)
37965
awe> frame = q[0]
awe> frame.filename
'Sci-WVRIEND-WFI-------#845-ccd56-----Sci-53628.6100091.fits'
awe> frame.raw.filename
'WFI.2000-12-20T01:21:43.805_7.fits'
awe> frame.bias.filename
'2000-04-26cal541_ccd56.fits'
awe> frame.flat.filename
'2000-04-21cal546-#845_ccd56.fits'
[1]The get_logs() method also can be invoked with a job ID (or a list of job IDs) as an argument.