Key concepts¶

Persistent classes¶

Astro-WISE data processing is performed by executing methods on instances of persistent classes (persistent objects). This means that all processing results are recorded in the persistent attributes of these objects, and the persistence mechanism ensures that these results are stored in the data base. (See Persistency Interfaces for further details)

targets, dependencies, make¶

At the highest level, Astro-WISE data processing can be understood in terms of targets, dependencies and make. To illustrate these concepts, let’s start with an example:

# example1.py
from astro.main import BiasFrame
from astro.main import RawFrame
 4
def makebias(raw_bias_names, bias_name):
  '''Make a master bias
  raw_bias_names -- a list of names of raw bias FITS files
  bias_name -- the name of the master bias FITS file
  '''
  bias = BiasFrame.BiasFrame(pathname=bias_name)
  for name in raw_bias_names:
      raw = RawFrame.RawBiasFrame(pathname=name)
      bias.raw_bias_frames.append(raw)
  bias.make()
  return bias

The example defines one function (makebias) to make a bias frame. It takes a list of the names of raw bias files and the name of the output file as arguments and returns a BiasFrame object. This example illustrates how a user would use the library to process his own data. For example, from the Python command line:

>>> from example1 import makebias
>>> bias = makebias(['ima01.fits', 'ima02.fits', 'ima03.fits'], 'bias.fits')

Let’s go over this piece of code line by line (note that Python source code does not include line-numbers):

lines 2-3: import two modules (BiasFrame and RawFrame) from the package astro.main
line 5: define a function (makebias) taking two arguments ( raw_bias_names and bias_name)
lines 6-9: the documentation for the function.
line 10: create a BiasFrame object. The BiasFrame class is defined in the BiasFrame module which we imported in line 1. The BiasFrame object is defined with one argument; the name of the bias image (bias_name)
line 11: loop over the names contained in the list raw_bias_name, assign each name to name
line 12: create a RawBiasFrame object from each name
line 13: add the raw bias object to list of input frames (called raw_bias_frames) of the master bias object bias
line 14: “make” the master bias object. By calling the method make, the processing necessary to create the master bias data is executed.
line 15: we are ready, and return the result.

this description probably doesn’t add much to your understanding of the example. If you don’t have the feeling that the description and the example are really equivalent, you should probably first try to get to know Python a little bit better.

The example illustrates the fundamental steps in processing data:

create the target object (line 10)
assign objects to the dependencies (lines 11-13)
execute the make-methods (line 14)

In this case the target (a BiasFrame object), only depends on the raw data (RawBiasFrame objects). In other cases the target may also depend on additional objects, including calibration data and processing parameters. For example, to reduce science data we need a considerable number of other objects besides the raw data, i.e. all calibration objects.

Note that a BiasFrame object is not a FITS file, it is an entity that describes a FITS file and may execute a number of operations on FITS files. All pipeline processing is done by calling methods on these kinds of objects.

The following methods can be used to inspect targets:

is_made(): return 1 if make() has been executed on a target and 0 otherwise. The value of the special attribute process_status is inspected to determine this.
set_made(): indicate that the target has been made. This is usually called from the make() method. The value of the special attribute process_status is updated to record this.
get_dependencies(): returns a list of attribute names on which the target depends.

The case of making a bias is probably the simplest example. Other examples can be found by looking at the other recipes in the directory filerecipes of the library. Have a look at these recipes, including the Bias recipe, to see that all look extremely similar to this example.

Verification and quality control¶

In order to verify the results of the data processing, makable objects (will) have verify(), compare(), and inspect() methods. These methods implement basic quality control mechanisms.

verify: The verify() method inspects the values of various attributes of the object to see if these are within the expected range for that object. The purpose of this method is mostly to perform sanity-checks on measured results. It is assumed that the required measurements (for example image statistics) are done during data reduction (i.e. while executing make()), and stored in persistent attributes.
compare: The compare() method is used for default trend analysis. This is done by comparing with a previous version of the same object (last weeks bias, for example). This may be as simple as comparing attribute values, but may also involve more complex computations (e.g., subtracting the two images, and analysing the residuals)
inspect: Visual inspection of the data remains a powerful tool in quality control. The inspect() method provides the mechanism to record the results of visual inspection for posterity.

The make() and quality control methods set a flag in the processing_status attribute to record if these methods have been run. The following methods are available to inspect the processing status of makable objects:

is_verified(): returns 1 if verify() has been succesfully executed, 0 otherwise.
is_compared(): returns 1 if compare() has been succesfully executed, 0 otherwise.
is_inspected(): returns 1 if inspect() has been succesfully executed, 0 otherwise.

The quality control methods record their results by setting quality control flags. These quality control flags are given in the class definition using the QCFlag() property. Flags are stored in the special attribute quality_flags, using bit masking. The following methods are available to inspect these flags:

get_qcflags(): This method returns a list of the names of all possible flags.
get_qcflags_set(): This method returns a list of those flags that have been set. If no flags have been set, this returns an empty list.

The following class defines two quality control flags and a verify method that may set these flags.

class MyScienceResult(DBObject, ProcessTarget):
    TOO_MANY_GALAXIES = QCFlag(0, 'This is a bad sign')
    TOO_FEW_STARS = QCFlag(1, 'This is a really bad sign')

    def verify(self):
        if self.galaxy_count > 1e6:
            self.TOO_MANY_GALAXIES = 1
        if self.star_count < 100:
            self.TOO_FEW_STARS = 1
        self.set_verified()

Here is an example session using this class

>>> m = MyScienceResult()
>>> m.galaxy_count = 10000
>>> m.star_count = 1000
>>> m.verify()
>>> m.is_ok()
1
>>> m.star_count = 10       # simulate a problem
>>> m.verify()
>>> m.is_ok()
0
>>> m.get_qcflags_set()
['TOO_FEW_STARS']
>>> m.galaxy_count = 10000000
>>> m.verify()
>>> m.TOO_MANY_GALAXIES
1
>>> m.get_qcflags_set()
['TOO_MANY_GALAXIES', 'TOO_FEW_STARS']
>>> m.quality_flags        # bits 0 and 1 were set
3