Astro-WISE basics

Setting up your environment

In order to start up the command-line interface, it is necessary to modify your shell environment so the AWE software can be found and the system can log you in to the database. The former is done using the module framework, the latter by creating a configuration file for AWE:

1. Edit your shell configuration file

   On the network of the Kapteyn Astronomical Institute:

   If you’re using “csh” or “bash” add the following line to the file :math:`sim`/.cshrc or :math:`sim`/.bashrc, respectively:

   module load awe
   

2. Obtain a database account and setup your :math:`sim`/.awe/Environment.cfg file

   It is necessary to obtain a database username and password in order to login to and write (“commit”) to the database. Please ask your local Astro-WISE DBA for an account.

   In your home directory make a directory .awe (starting with a dot) and in this subdirectory make a file called “Environment.cfg”, readable only to yourself:

   ...]$ cd
   ...]$ mkdir .awe
   ...]$ cd .awe
   ...]$ touch Environment.cfg
   ...]$ chmod a-rwx,u+rw Environment.cfg
   

   Then add the following lines to that file replacing your username (e.g., awjkennedy) and password (e.g., IwonIn1960) in the appropriate places. The line project ensures the awe-prompt always starts in context ALL (what context is will be explained later in the tutorial). The line privileges : 1 ensures that it starts within the private space of that context (‘MyDB’).

   [global]
   database_user : awjkennedy
   database_password : IwonIn1960
   project : ALL
   privileges : 1
   

   Start the awe-prompt from your shell:

   ...]$ awe
   

   Check that your username and default project are used. In the above case that means the welcome message ends with a message like this:

   Current profile:
   - username : awjkennedy
   - database : db.astro.rug.astro-wise.org
   - project  : ALL
   - current privileges : 1 (MyDB)
   
   awe>
   

   you now have access to the database. Quit awe by hitting Ctrl-d.

At the awe-prompt: Looking Around

1. Start the awe-prompt  by typing awe at the command line. The awe-prompt  is the standard Python interpreter plus a few additions. The Python interpreter has built-in functions and classes:

   awe> dir(__builtins__)
   

   If you are not familiar with the Python language and/or object oriented programming it is a worthwile to browse this Python tutorial. It will make the rest of the tutorial much easier to grasp. Many object oriented programming concepts (especially the chapter on classes) are crucial concepts at the awe-prompt. The built-in functions of Python contain basic functions such as “range”, “len”, “int”, “float”, “dir” and “help” that can be useful at the awe-prompt. To illustrate this, the following example calculates element 19 of the Fibonacci sequence using Pythonsyntax:

   awe> int(round( ( (1.0+5.0**0.5)**19 - (1.0-5.0**0.5)**19 ) / (2.0**19*5.0**0.5) ))
   
   4181
   

2. Conventional subjects in astronomy are represented at the awe-prompt by (Python) classes. These may be images (e.g., masterflats, raw science images, coadded frames) and catalogs, or instruments, chips, astrometric solutions, photometric solutions etc. Conventional metadata such as found in FITS headers are associated with these classes as properties. Examples are exposure time, observation date, as well as links to other objects that were used to create the object. Methods are functions associated with the class. For example, there are methods named commit, make, inspect and do_statistics. The most commonly used classes are automatically loaded when starting the AWEenvironment and their names are listed in the main “namespace”. You can print the main namespace by entering:

   awe> dir()
   

3. Start the built-in help system in a similar way:

   awe> help()
   
   awe> help()
   
   Welcome to Python 3.5's help utility!
   
   If this is your first time using Python, you should definitely check out
   the tutorial on the Internet at http://docs.python.org/3.5/tutorial/.
   .
   .
   help>
   

   Follow the instructions to find help on pretty much anything by typing the name of it. Use quit or Ctrl-d to exit. It is usually convenient to get help on a specific module, function, or even object. To see what dir() does exactly, for instance, type:

   awe> help(dir)
   # the screen clears and shows the following:
   Help on built-in function dir in module __builtin__:
   
   dir(...)
       dir([object]) -> list of strings
   
       If called without an argument, return the names in the current scope.
   .
   .
   

4. One of the classes listed is called Chip. This is a representation of a CCD. It has its own namespace which can be printed by typing:

   awe> dir(Chip)
   ['__class__', '__del__', '__delattr__', '__dict__', '__dir__', '__doc__',
   '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__',
   '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__',
   '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__',
   '__str__', '__subclasshook__', '__weakref__', '_delete', '_inverses',
   '_publish', 'as_dict', 'commit', 'copy_persistent_properties', 'dbproxy',
   'get_creator', 'get_inverse_properties', 'get_persistent', 'get_project',
   'info', 'inverse_objects', 'inverse_query', 'name', 'object_id', 'persists',
   'pickle_id', 'pixelsize', 'recommit', 'select_all', 'update_header']
   

5. Another class is called RawTwilightFlatFrame. It is a representation of a raw sky flat. It also has its own namespace:

   awe> dir(RawTwilightFlatFrame)
   ['DATE', 'DATE_OBS', 'EXPTIME', 'LST', 'MEAN_HIGH', 'MEAN_LOW', 'MJD_OBS',
   'NAXIS1', 'NAXIS2', 'OBJECT', 'OBSERVER', 'OVSCX', 'OVSCXPRE', 'OVSCXPST',
   'OVSCY', 'OVSCYPRE', 'OVSCYPST', 'PROCESS_TIME', 'PRSCX', 'PRSCXPRE',
   'PRSCXPST', 'PRSCY', 'PRSCYPRE', 'PRSCYPST', 'STATUS_COMPARE',
   'STATUS_INSPECT', 'STATUS_MAKE', 'STATUS_VERIFY', 'UTC', '_IS_ABSTRACT',
   '_IS_CAL', '_IS_CONFIG', '_IS_RAW', '_IS_SCIENCE', '_IS_SEQ', '_IS_SUPPORT',
   .
   .
   

   The namespace lists both properties (e.g., observation date DATE_OBS, exposure time EXPTIME) and methods (e.g., inspect, display) of the class.

6. Telescope calibration files such as sky flat-fields have important properties that are stored in the Astro-WISE database. These are called persistent properties and they can be listed as follows:

   ['DATE', 'DATE_OBS', 'EXPTIME', 'LST', 'MJD_OBS', 'NAXIS1', 'NAXIS2', 'OBJECT',
   'OBSERVER', 'OVSCX', 'OVSCXPRE', 'OVSCXPST', 'OVSCY', 'OVSCYPRE', 'OVSCYPST',
   'PRSCX', 'PRSCXPRE', 'PRSCXPST', 'PRSCY', 'PRSCYPRE', 'PRSCYPST', 'UTC',
   'chip', 'creation_date', 'extension', 'filename', 'filter', 'globalname',
   'imstat', 'instrument', 'is_valid', 'object_id', 'observing_block',
   'overscan_x_stat', 'overscan_y_stat', 'prescan_x_stat', 'prescan_y_stat',
   'process_params', 'process_status', 'quality_flags', 'raw_fits_data',
   'template']
   

   In this list “chip” points to another class: the class Chip we just described. To list its persistent properties type:

   awe> RawTwilightFlatFrame.chip.get_persistent_properties()
   ['name', 'object_id']
   
   or
   
   awe> Chip.get_persistent_properties()
   ['name', 'object_id']
   

7. You can find out what each property means by entering:

   awe> help(RawTwilightFlatFrame)
   Help on class RawTwilightFlatFrame in module astro.main.RawFrame:
   
   class RawTwilightFlatFrame(RawFrame)
    |  A base class for persistent FITS file objects
    |
    |  BaseFrame inherits from the persistent DataObject class and the
    |  ProcessTarget mixin. DataObject provides data retrieval and
    |  storage facilities for the FITS files. All BaseFrame objects are
    |  ProcessTargets (have a make method).
    |
    |  BaseFrame itself is intended to be an abstract baseclass.
    |
    |  A BaseFrame instance defines a number of attributes,
    |
    |  image   -- An eclipse.image object (default None), read if needed,
    |             or by load_image()
    |  header  -- An darma.header object (default None), read if needed,
    |             or by load_header(), or created by build_header()
    |  history -- A list of strings containing the history of the object.
    |             currently not read from the header, but used by
    |             build_header()
    |
    |  Method resolution order:
    |      RawTwilightFlatFrame
    |      RawFrame
    |      astro.main.BaseFrame.BaseFrame
    |      common.database.DataObject.DataObject
    |      common.database.DBMain.DBObject
    |      astro.main.ProcessTarget.ProcessTarget
    |      common.database.DBMeta.DBMixin
    |      common.main.ProcessTarget.ProcessTarget
    |      astro.main.OnTheFly.OnTheFly
    |      common.main.OnTheFly.OnTheFly
    |      builtins.object
   ...
   ...
    |  ----------------------------------------------------------------------
    |  Data descriptors defined here:
    |
    |  DATE
    |      UTC date the original data file was saved [None]
    |
    |  DATE_OBS
    |      UTC date at the start of the observation [None]
    |
    |  EXPTIME
    |      Total observation time [sec]
   .....
   
   Hit "q" to exit the help page.
   

   Here the difference between DATE and DATE_OBS becomes clear. The help tells you much more than just the properties. The first line tells you that the code for this class is stored in module astro.main.RawFrame (how to find the code will be explained later). It is followed by a definition of RawTwilightFlatFrames. The explanation of the other lines is beyond the scope of this tutorial.

8. You can use the persistent properties to query the database for a particular RawTwilightFrame. Lets find all OmegaCAM RawTwilightFlatFrames which were observed in August 2011 (the start of science observations with OmegaCAM):

   NB: enter the following 2 lines on 1 line of the awe prompt:
   
   awe> query = (RawTwilightFlatFrame.instrument.name == 'OMEGACAM') &
   (RawTwilightFlatFrame.DATE_OBS < datetime.datetime(2011,9,1))
   
   NB: you can enter the following command on 2 input lines by continuing
       the line with '\':
   
   awe> query = (RawTwilightFlatFrame.instrument.name == 'OMEGACAM') & \
                (RawTwilightFlatFrame.DATE_OBS < datetime.datetime(2011,9,1))
   
   or by using an extra set of parentheses:
   
   awe> query = ((RawTwilightFlatFrame.instrument.name == 'OMEGACAM') &
   (RawTwilightFlatFrame.DATE_OBS < datetime.datetime(2011,9,1)))
   
   awe> len(query)
   6240
   

   You can select one of these RawTwilightFrames using an index and see when it was observed exactly:

   awe> raw = query[10]
   awe> raw.DATE_OBS
   datetime.datetime(2011, 8, 10, 22, 49, 22)
   

   or more directly:

   awe> query[10].DATE_OBS
   datetime.datetime(2011, 8, 10, 22, 49, 22)
   

   and retrieve the image from the dataserver (download to your current working directory):

   awe> raw.retrieve()
   

   once retrieved, you can display it:

   awe> raw.display()
   

   or inspect it:

   awe> raw.inspect()
   

   The last three commands are method calls. Methods typically perform an operation on a (Python) object. The object “raw” is called an instantation of the class RawTwilightFlatFrame. The last command brings up a graphical window: enter ‘q’ when you hover over it to quit the graphical window.

9. One other very useful method on instances of database objects in AWE, is info(). This method prints out all the persistent attributes and their values:

   awe> chip = Chip()
   awe> chip.info()
   Chip: <astro.main.Chip.Chip object at 0xb03cd0ec>
    |
    +-name:
    +-object_id: '000000000000000000000000000000000
   

   The Chip class has only one useful persistent attribute: name. object_id is a unique object identifier identifying each object in the database. Try to run the info method on all the other objects in these tutorials to get their information. Be aware, however, that not all objects have this method.

   Notes:

   • For this tutorial it can be helpful to use “Tab” key a lot for automatic completion of names. Tab completion does not work on queries of objects (queries are explained in next section).
   • A command-line history is saved. Using the “Up”, “Down” or “Ctrl-p”, “Ctrl-n” keys you can recall previous commands. If you specify a partial command the history will be searched for matches which start with the same characters.
   • Make sure you use the class when using get_persistent_properties() (i.e., ) and not an instance of the class (i.e., raw.get_persistent_properties()).

The power of querying

1. Which instruments are stored in the database?

   This can be done in several ways. One way is to query for all instruments of which the name is not equal to an empty string. Alternatively you could use the “like” functionality to ask for all Instruments for which the name is equal to any string.

   awe> q = Instrument.name != ''
   awe> for i in q: print(i.name)
   
   awe> q = Instrument.name.like('*')
   awe> for i in q: print(i.name)
   

   Reminder: Instrument.get_persistent_properties() shows you that Instrument has the property name stored in the Astro-WISE database. Entering:

   awe> Instrument.n
   

   and hitting the Tab key also reveals that Instrument has the property name. NB: Tab completion does not work on queries (i.e., q.n following above example).

2. What are the names of the chips (CCDs) present in the OmegaCAM camera?

   You can ask this of the context object, instantiated by default at the AWE prompt:

   awe> context.get_chips_for_instrument('OMEGACAM')
   

3. How many RawScienceFrames for OmegaCAM are stored in the Astro-WISE database?

   awe> q = (RawScienceFrame.instrument.name == 'OMEGACAM')
   awe> len(q)
   6592993
   

   …and counting. Please note that for each OmegaCAM exposure 32 (one for each CCD) RawScienceFrames are created in the database.

4. Execute these lines at the prompt to select a particular ReducedScienceFrame (a flat-fielded and de-biased science image, for a single CCD) from our database.

   awe> sci = ((ReducedScienceFrame.DATE_OBS == datetime.datetime(2011, 8, 6, 8, 5, 20)) & \
               (ReducedScienceFrame.chip.name == 'ESO_CCD_#65')).min('creation_date')
   

   For this ReducedScienceFrame, retrieve from the Astro-WISE dataserver the image itself and the bias image that was used. The FITS images will be saved to the current working directory.

   awe> sci.retrieve()
   awe> sci.bias.retrieve()
   

   The images, such as ReducedScienceFrames, which you and others create in Astro-WISE are stored on the Astro-WISE dataservers. As explained in more detail later, this will only be done if you explicitly commit these images to Astro-WISE. Even after committing them, you can invalidate images at any time if you decide they are no good after all.

5. For the ReducedScienceFrame of the previous exercise print the observing date, the object, instrument, filter and chip (CCD) name.

   (NB: enter following 2 lines as one line on the awe prompt.)
   awe> print(sci.DATE_OBS, sci.OBJECT, sci.instrument.name, sci.filter.name,
   sci.chip.name)
   # Or
   awe> print(sci.DATE_OBS)
   awe> print(sci.OBJECT)
   # etc.
   

6. For the ReducedScienceFrame selected, print the observation date DATE_OBS of the RawBiasFrames used in the creation of the master bias that was used in debiasing the ReducedScienceFrame.

   awe> for frame in sci.bias.raw_bias_frames: print(frame.DATE_OBS)
   

   And to print the equivalent modified Julian date:

   awe> for frame in sci.bias.raw_bias_frames: print(frame.MJD_OBS)
   
   or by calculating from the DATE_OBS:
   
   awe> from common.util.utilities import datetime_to_mjd
   awe> for frame in sci.bias.raw_bias_frames: print(datetime_to_mjd(frame.DATE_OBS))
   

   The “from … import …” statement loads a method to do the conversion. To find out what can be imported from a module, simply import the module and use the built-in dir() function:

   awe> from common.util import utilities
   awe> dir(utilities)
   

   To get help on the entire module, use the built-in help() function:

   awe> help(utilities)
   

7. What were the exposure levels of the raw dome flats that were used?

   Note that the flat-field used in the reduction is a MasterFlatFrame which is this case was created from a master dome (DomeFlatFrame).

   (NB: make sure to indent the 2nd line below:)
   awe> for frame in sci.flat.domeflat.raw_domeflat_frames: print(frame.imstat.median)
   

More Advanced Queries

1. Find out how many RawBiasFrames there are for the OmegaCAM instrument where the bias level is greater than 1000 ADU.

   The property of the RawBiasFrame that contains its image statistics is called “imstat”. This is an instance of the “Imstat” class, which itself has properties such as “mean”, “median” and “stdev”. We need to combine this query with a query on the name of the instrument of the RawBiasFrame.

   NB: print following lines on a single awe prompt line.
   awe> q = (RawBiasFrame.instrument.name == 'OMEGACAM') & \
            (RawBiasFrame.imstat.median > 1000.0)
   awe> len(q)
   

2. Find out how many RawScienceFrames observed the first week of November 2011 are present in the database for the OmegaCAM instrument. For these RawScienceFrames, print the observation date, the filter, the R.A. and Dec, and the OBJECT header keyword.

   Here a lot of things come together. Usually, frame attributes that are all upper-case letters correspond to a FITS header keyword of the same name. This will apply to only some of our attributes. We will need to know the names of the other attributes that we are interested in, and if they can be queried on at all. This is done as follows:

   awe> RawScienceFrame.get_persistent_properties()
   

   In particular note the DATE_OBS, “astrom” and OBJECT properties. The R.A. and Dec can be found in “astrom” (as CRVAL1 and CRVAL2, respectively). In addition, we are directly querying on a datetime object (DATE_OBS) in order to get the data for 2011. This requires that you create a datetime object for your dates, so you can compare the two.

   It may be helpful to limit the query to a single CCD, in order to avoid getting 32 times the same information (once for each CCD of the OmegaCAM camera).

   awe> dat1 = datetime.datetime(2011,11,1)
   awe> dat2 = datetime.datetime(2011,12,1)
   awe> q = (RawScienceFrame.DATE_OBS>dat1) & \
            (RawScienceFrame.DATE_OBS<dat2) & \
            (RawScienceFrame.chip.name=='ESO_CCD_#65') & \
            (RawScienceFrame.instrument.name=='OMEGACAM')
   awe> for s in q: print(s.DATE_OBS, s.filter.name, s.OBJECT, s.astrom.CRVAL1,
   s.astrom.CRVAL2)
   

System Calls from the awe-prompt

1. It is possible to do Unix/Linux system calls from the awe-prompt. The “os” module is imported by default and can be used to run commands as follows:

   awe> os.system('ls')
   awe> os.system('pwd')
   

   awe> command = 'skycat file.fits&'
   awe> os.system(command)
   

   To move one directory up use:

   awe> os.chdir('..')
   

Understanding Python errors/exceptions/backtrace

1. If you type something erroneous at the awe-prompt an error message will be returned. The last sentence of the message is almost always the most useful line to determine the cause of the error.

   awe> RawScienceFrame.instrument.ThisAttributeDoesNotExist=='bla'
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/Software/users/astro-wise/awehome35/AWBASE/common/database/DBProperties.py", line 226, in __getattr__
       persistent_property.__getattr__(self, attr)
     File "/Software/users/astro-wise/awehome35/AWBASE/common/database/DBProperties.py", line 140, in __getattr__
       self.attribute, self.cls.__name__, attr))
   AttributeError: Persistent attribute "instrument" of class "RawScienceFrame" does not have a persistent attribute "ThisAttributeDoesNotExist"