HOW-TO Introduction

History

The Astro-WISE Environment (AWE) was created by The Astro-WISE Consortium, a partnership of:

and coordinated by OmegaCEN-NOVA. It was conceived as the solution to handle the vast amounts of astronomical data generated by all-sky surveys, particularly those to be observed with OmegaCAM on the VLT Survey Telescope (VST) on Cerro Paranal in Chile.

While waiting for OmegaCAM and VST completion, AWE has been expanded to include data from many different sources. Currently supported data sources include the Wide-Field Imager (WFI) on the 2.2m MPG at La Silla, the Wide-Field Camera (WFC) on the INT at La Palma, and SuprimeCAM (SUP) on Subaru at Mauna Kea. The public portions of these datasets can be browsed here.

Basic Philosophy

  • General
    • This is an information system for the scientific analysis of extremely large datasets. It utilizes federated databases and dataservers, and parallel compute clusters to manage these vast amounts of data.
    • It was originally designed and developed specifically for astronomical wide-field imaging surveys, but has been used for the analysis of large datasets of handwritten archives and can be applied to any application involving very large datasets.
    • AWE is a federated system: data can be made any one-location in the federated system, but used everywhere in the federated system. This allows collaboration between diverse working groups. AWE is currently operational in The Netherlands (Groningen and Leiden), Germany (Bonn and Munich), and Italy (Naples)
    • Raw data is sacred. All data in the system is derived from raw data and can be traced back to the raw data within the system. This facilitates on-the-fly re-processing whenever improved methods or data are available.
    • All data is linked via backward chaining. Thus system is an all-in-one system: ALL input and output of processes are stored in the information system.
    • The most recent product of a series of processes is considered the best: newer is better. Processes automatically use the latest versions of calibration files and software codes, which are both assumed to be the best.
    • The processing of data is split up such a way that it is embarassingly parallel. For astronomical data, this means all processes are per CCD.
    • AWE makes extensive use of the Python programming/scripting language.
  • Object Model
    • Data are represented as Python objects with attributes corresponding to both pixel-data and meta-data, and methods corresponding to actions to perform on the object and its attributes.
    • Each object is considered a target that can be made with Tasks (e.g., found in the astro/recipes directory ) which set mandatory dependencies and run the target’s make() method. Every make() fills in the newly instantiated (i.e., created) object. For example: awe> bias = BiasFrame() instantiates an EMPTY bias frame object. awe> bias.make() completes/fills in the object’s attributes such as the observation date, pixel and header data, data statistics, etc.
    • With the Target Processor, the concept of a UNIX “make” is duplicated where all dependencies are checked for up-to-dateness (this includes existence) and will have their own make() method run if they are not up-to-date. This occurs recursively back to the raw data. NOTE: This recursive “making” of objects does not extend to the Tasks as the dependencies are only checked for existence.
    • Each object has verify(), inspect(), and compare() methods. These ensure optimum data quality.
  • Code Access
    • Users have direct access to the Python code base via Git checkout at https://gitlab.astro-wise.org/
    • Users have the ability to add their own code or modify certain existing code.
    • The core parts of the system should not be modified by users.
    • Recipes can be modified for convenience in the users own checkout.
    • All the other parts of the system should only be modified if there is a bug, and preferably by the maintaniner of that part.
    • See Python and Object Oriented Programming for more information.
  • Documentation
    • The Astro-WISE portal (http://portal.astro-wise.org/) is the primary location of documentation for AWE.
    • The HOW-TOs are the main source of documentation for operating the system. They describe all aspects of data processing in AWE, contain tutorials and troubleshooting guides, etc.
    • There are two mailing lists: Issues and News
    • Documentation for the code can be found at http://doc.astro-wise.org/ or from the AWE prompt via Python’s help() function.
  • Services
    • See Web Services for the various web-based interfaces to the system.

Hardware

In our architecture, three main components are identified: a file-server, a database, and a high-performance compute cluster.

The file-server stores FITS-files, while the database keeps track of the relations between these files and their processing history. It is also through this database that decisions are made about which files to retrieve during the various processing steps. The compute cluster is used to process the data. During processing, requests are made to the database for the raw science data and for the necessary calibration files, which are then retrieved from the file-server.

NOTE: The only files the user has direct access to locally (e.g., current directory) are data products retrieved and/or created during processing on the local machine. Normal processing using the compute cluster will leave no such data products, but only log files at most.

Software

The software consists of code written in Python, and includes an interactive command line environment (see HOW-TO Use the awe-prompt) along with web services for viewing and editing the database (see Web Services). On a lower level a number of existing C programs are used, such as SExtractor, LDAC, SWarp, and Eclipse. (see Interfaces to other programs).

Python and Object Oriented Programming

The code for the Astro-WISE system is written in Python, which is a language highly suitable for Object Oriented Programming (OOP). Within the OOP style in which the code is written, classes are associated with the various conventional calibration images, data images, and other derived data products. For example, in our system, bias exposures become instances of the RawBiasFrame class, and twilight flats become instances of the RawTwilightFlatFrame class. These instances of classes are the “objects” of OOP.

Classes may have incorporated methods and attributes. Methods perform a task on the object they belong to, while attributes are properties such as constants, flags, or links to other objects that may be needed by methods. In Astro-WISE various recipes have been coded that control the creation of instances of these classes. There may be different ways to create an instance of a class depending on which attributes are set to what values, and which methods are used. A ColdPixelMap object, for example, can be instantiated from the database (i.e. as the result of a query or search) or it can be created by using its “make()” method. In the latter case the ColdPixelMap can be derived either from a DomeFlatFrame or a TwilightFlatFrame, depending on which of those two objects are specified as the “flat” attribute of the ColdPixelMap object.

Within the Object Oriented Programming style, inheritance is an important and powerful concept. Classes can inherit attributes and methods from other classes. For example, both the DomeFlatFrame and TwilightFlatFrame classes are derived from the base class BaseFlatFrame; both are flat-fields afterall. Any method or attribute defined by the BaseFlatFrame class is inherited by both classes and both classes are free to redefine (this is called polymorphism) those methods or attributes and even add new ones as needed.

The bottom line is that Astro-WISE turns data into objects that are instances of Python classes with attributes and methods that can be inherited.

There is a significant amount of on-line documentation available for Python. Please see the Python web site http://www.python.org for further background on Python and for Python tutorials.

Web Services

Database Viewer

A web-service is available to view the database content and pixel data. It can be found at the following address:

http://dbview.astro-wise.org/

Help pages are provided by the webservice itself.

Database “Editor”

For a limited number of operations editing of database values is allowed. Specifically, it is possible to change valid ranges of calibration data (timestamps), and flags, to disqualify bad data. A special web-service tool to change these values can be found at the following web-site:

http://calts.astro-wise.org/

Help pages are provided by the webservice itself.

Processing Web Interface

The Target Processor is the culmination of all the benefits of the Astro-WISEsystem. It allows processing of a target (an end data product such as a ReducedScienceFrame or SourceList) and any of its dependencies that require it, on a parallel (e.g., compute cluster) or on a single (e.g., local machine) host. The dependency chain is followed back towards the raw data (backward chaining) to make sure only those objects requiring processing are actually processed.

http://process.astro-wise.org/

A web-based version of the awe-prompt command-line interface also exists for convenience (account required).

http://process.astro-wise.org/AWE

Image Handling Services

Services involving image sections (dependency cutouts, RGB generator, etc.) can be found in the IMGView service:

http://imageview.astro-wise.org/

Further Websites

The OmegaCAM web site:

http://www.astro.rug.nl/~omegacam/

The Astro-WISE web site:

http://www.astro-wise.org/

The Astro-WISE Web Services web site:

http://portal.astro-wise.org/