HOW-TO Use Astro-WISE Context to efficiently process, qualify, publish and manage project data

The ocean of data objects in Astro-WISE is organized by providing them with attributes with information about projects, share-levels (called privileges), validity (set exclusively by users) and quality flags (set esclusively by the system).

Via the Astro-WISE concept Context you can interact with these attributes projects, share levels, validity and system quality_flags to efficiently perform the main project tasks:

  • data processing
  • validating data
  • publishing of project data
  • project management

See the HOW-TO Use your Astro-WISE Context to set data scopes for a disussion on projects, privileges and users.

An Astro-WISE user can set 3 levels of validity of an object using its is_valid attribute:

  • 0: the object is invalid
  • 1: the object is valid
  • 2: the object is proposed to be promoted to the next share-level

A user can retrieve the value of the is_valid attribute for the data object myobject on the awe command line as:

awe> myobject.is_valid
2

For calibration data, the user can also set the time range of validity of an calibration data object using the attributes timestamp_start and timestamp_end. For example, a masterflatfield might be valid for a week centered on its observation date.

How timestamps are exploited in Astro-WISE is explained in HOW-TO Work with dates and times in AWE.

A user can retrieve the value of the timestamps of the data object mycalibrationobject on the awe command line as:

awe> mycalibrationobject.timestamp_start,mycalibrationobject.timestamp_end
(datetime.datetime(2001, 1, 1, 16, 0), datetime.datetime(2001, 12, 31, 16, 0))

The Astro-WISE system sets quality flags for each data object upon its creation via the attribute quality_flags. A quality_flag=0 indicates that the system did not detect any problems in terms of quality of the data while a quality_flags>0 does. [TBWritten Ewout: document with descriptions of values of ``quality_flags`` and what they mean which can be referenced to from here.] The value of quality_flags cannot be changed by users.

Via the Astro-WISE Context you can interact with the matrix of projects, share-levels, validity and system quality_flags via these actions:

  1. data scope: setting the projects and share-levels accessible in your working environment (in other words, the projects and share-levels from which data will be drawn for your processing)
  2. validating: updating the validity of objects
  3. publishing: changing the share-level of data objects

Next we discuss these 3 actions.

Data scope using Context

See the HOW-TO Use your Astro-WISE Context to set data scopes for howto set projects and privileges.

Data creation

Data objects which you create will belong to the current project and current share-level. Let us assume that the project set to SONATE and share-level (called privileges) to 2. Newly created data will then become part of SONATE and visible to the project members of SONATE.

Because timestamps, validity and quality flags of calibration data are stored in the Astro-WISE database it is possible for the Astro-WISE system to determine which calibration data should be used when a user processes data. The minimal requirement for calibration data to be nominated for selection are:

  • timestamp interval includes the observation date of the science data
  • quality_flags=0 AND is_valid>0

By default the selection procedure goes according to the following criteria in order of decreasing importance:

  • calibration data in the current project is favored over data from other projects
  • calibration data at lower share-levels is favored over higher share-levels
  • calibration data created most recently created is preferred over older data

This results in the following decision tree for each calibration data object individually:

  1. Is the calibration data present at share-level 1 in the current project?:
    • yes: use the one most recently created
    • no: go to next decision
  2. Is the calibration data present at share-level 2 in the current project?:
    • yes: use the one most recently created
    • no: go to next decision
  3. Is the calibration data present at share-level 3,4 or 5 in the current project?:
    • yes: use the one most recently created
    • no: go to next decision
  4. use the most recently created from share-levels 3,4,5 of all projects

Both the webservice for processing, TargetProcessor and the awe prompt use this default selection procedure. The default selection behavior can be overruled at the awe prompt. This can be done for example by selecting the required calibration files via your own queries. See Section xx for details on queries.

Data ingestion

Ingestion of observations into Astro-WISE is described in HOW-TO Ingest raw data into the database.

The default is that calibration observations are ingested at share-level=4 and science observations are ingested at share-level=1. [TBD John: implement above default behavior]

Validation using Context

At the awe prompt, Astro-WISE users can modify the validity of data object myobject by changing the value of its attribute is_valid to new_value:

awe> context.update_is_valid(myobject, new_value)

An Astro-WISE user can set 3 levels of validity of an object using its is_valid attribute:

  • 0: the object is invalid
  • 1: the object is valid
  • 2: the object is proposed to be promoted to the next share-level

By default is_valid=1. The is_valid is strictly a validation by the Astro-WISE user. Only an Astro-WISE user changes the value of is_valid, not the system. The system sets the quality_flags.

At the awe prompt, Astro-WISE users can modify the validity time range of a calibration data object mycalibrationobject to the period from 2001, February 13 16:00 UT to 2001, February 20 13:00 UT by changing its attributes timestamp_start and timestamp_end

awe> context.update_timestamp_start(mycalibrationobject},datetime.datetime(2001, 2, 13, 16, 0))
awe> context.update_timestamp_end(mycalibrationobject},datetime.datetime(2001, 2, 20, 16, 0))

A calibration object obtains a default validity time range around it observation date upon creation. See section xx for a detailed description of these.

Updating attributes of objects is restricted, for the different privileges the following rules apply :

  • privileges 1 : only the creator is authorized to update
  • privileges \(>=\) 2 (private projects): only the project manager is authorized to update
  • privileges \(>=\) 2 (public projects): the project manager and the creator are authorized to update

For calibration data, the webservice CalTS provides a graphical interface to modify timestamps and is_valid. See the help section of CalTS for instructions. For science data, the webservice QualityView provides a graphical interface to do this. This webservice can be accessed via the QuickDataSearch under the column “quality” in the results form. It can also be accessed via the webservice DBViewer via the button “QualityView” in the column obj_ID in the tabular results forms of science images. [TBD: John: scope work for stand alone access]

In both CalTS and QualityView the user can attach textual comments to a data object. These comments can be read by the project manager and other Astro-WISE users. The free format allows project teams to internally organize systematic usage of these to qualify data to their specific needs.

Thus multiple people in project teams can validate project data. The idea is that the project manager verifies the validation by the team at share-levels \(\geq 2\). Note that single share-levels and projects can be isolated via queries:

awe> q = ReducedScienceFrame.is_valid == 2
awe> len(q)
3753
awe> q = q.privileges_only(3)
<common.database.DBOracleSelect.Select object at 0x93ae26c>
awe> len(q)
437
awe> q = q.project_only()
awe> len(q)
200

Based on his inspection the project manager can decide to promote data objects to a more public share-level. Such ‘publishing’ of data is discussed below.

Deleting objects

See the HOW-TO Use your Astro-WISE Context to set data scopes for howto delete objects using context.

Publishing using context

See the HOW-TO Use your Astro-WISE Context to set data scopes for howto publish objects using context.