HOW-TO Use your Astro-WISE Context to set data scopes

The Astro-WISE concept Context allows you to filter from the ocean of data objects in Astro-WISE the subset of data objects that you want to be currently visible to you and your processes. This is the ‘data access scope’ of your Context. At the same time your chosen Context also defines the logical subset to which the results of your processes will belong. This is the ‘data creation scope’ of your current Context. In other words, by configuring your Astro-WISE Context you define the logical subspaces in the ensemble of data objects in Astro-WISE in which you access data and in which you create data.

Astro-WISE Context

Your Astro-WISE Context is fully defined by choosing the settings of three Astro-WISE entities:

  1. Astro-WISE user identity
  2. project
  3. minimum privileges level

User

An Astro-WISE user is a person with an Astro-WISE database account. An Astro-WISE user has an id number and name (e.g., AWJJOPLIN). Each person has only one account. Thus, each person has a single identity within the Astro-WISE system. Each data object is associated with a single Astro-WISE user, the data object creator. This is the Astro-WISE user that created/ingested the data. Once established the creator of a data object cannot be changed.

At the awe-prompt the creator of data object myobject can be printed to the screen with

awe> myobject._creator # Returns user id
990
awe> from common.database.Database import database
awe> Database.users[myobject._creator] # Returns user name
'AWJJOPLIN'

Project

A project in Astro-WISE is a set of Astro-WISE users who collaborate on a common data set. A project has a project id, a name, a description, project members and optionally an instrument. One or more Astro-WISE users can be member of a project, and an Astro-WISE user can be member of more than one project as illustrated in Figure 1. Unlike an user each data object belongs to one and only one project. The project to which a data object belongs is chosen upon the creation/ingestion of the data entity and can not be changed after that.

_images/ProjectsDiagram.png

Figure 1: This diagram shows which Astro-WISE users are project members of Astro-WISE project A and B. A project in Astro-WISE is a set of Astro-WISE users who collaborate on a common data set. Project members are indicated in the diagram by a green tick mark. An Astro-WISE user can be member of more than one project. Each project has one or more Astro-WISE users as project members. One or more of the project members can be project manager (indicated with (pm)). A project manager has additional abilities and responsibilities to manage project data.

At the awe-prompt the project to which data object myobject belongs can be printed to the screen with

awe> myobject._project #Returns project id
14
awe> from common.database.Database import database
awe> Database.projects[myobject._project] # Returns project name
'WFI@2.2m'

Some projects have all Astro-WISE users as members: these are called public projects. Some projects have a subset of Astro-WISE users as members: these are called private projects. One or more of the Astro-WISE users in a project can also act as project manager. A project manager has additional abilities and responsibilities to manage data objects that belong to a project. The extra abilities/responsibilities are for data objects at privileges levels of 3 and larger. At these privileges levels the data objects are visible to persons which are not project members. The concept of privileges is explained next.

An overview of all projects in the database, their members, the project’s manager etc. can be found on our webpages at the following address: http://process.astro-wise.org/Projects.

Privileges

Privileges of data objects determine which Astro-WISE users have access to the data object. Each data object in Astro-WISE has one of five privileges as listed in Table 1.

Table 1: Privilege levels of Astro-WISE

privileges level data is shared with
1: MYDB only the creator
2: PROJECT every member of the project to which the data object belongs
3: ASTRO-WISE all Astro-WISE users
4: WORLD the world: Astro-WISE users and persons without an Astro-WISE account
  (latter via webservice DBViewer)
5: VO again the whole world and data also available through the Virtual Observatory
  via Virtual Observatory webservices

The initial privileges of a data object are set upon the creation/ingestion of the data entity. They can be changed after that.

At the awe-prompt the privileges of data object myobject can be printed to the screen with

awe> myobject._privileges
1

Using Context

Your Astro-WISE Context is defined by your Astro-WISE user identity, the current project you choose and the current minimum privileges level that you choose. The Astro-WISE user that you are is defined upon login as you login with a certain user name (e.g. AWJJOPLIN). Upon login, also a current project and a current minimum privileges level are set. These two initial selections depend on your login configuration. At the Astro-WISE command-line prompt, the python class context is the interface to view and change the selection of project and privileges level. You can import context at the Astro-WISE prompt via:

awe> from common.database.Context import context

To see which minimum privileges level is currently active use context.get_current_privileges()

awe> context.get_current_privileges()
1

To change the minimum privileges level call the context.set_privileges(<privileges>) method:

awe> context.set_privileges(2)

To see which project is currently selected, use context.get_current_project(). If the project object is printed, a formatted overview will be given:

awe> print(context.get_current_project())
   Project:     SONATE  (id = 14)
   Description: 1700 Square degree 5 band survey of the equatorial strip and 2dF
                South region
   Instrument:  OCAM
   Maximum Privileges: 4 (WORLD)
   Current Privileges: 1 (MyDB)

Instrument lists the instrument associated with the project. If it is None it means data from any instrument can be associated with the project. Maximum Privileges is the maximum privileges level which a project member can assign to a data object. To change to a project called ‘BLUES’ at the awe-prompt use:

awe> context.set_project('BLUES')

An user can only set projects of which the user is a member. When a project is selected without specifying the minimum privileges level they will be set to their default: 1.

To select at the same time the project and the minimum privileges level at the awe-prompt use:

awe> context.set_project('BLUES', privileges=2)

Data access scope

The philosophy in Astro-WISE is to increase efficiency in survey processing by sharing useful public data (such as calibration data). The philosophy of Context is therefore to allow users to select which data they want to see in their data access scope in addition to public data from Astro-WISE projects.

Whether a data object falls within the data access scope of the Context as currently configured depends on the values of 3 attributes which each data object has:

  • _privileges
  • _project
  • _creator

These attributes are compared to the corresponding 3 entities which configure your Context: the chosen values for project, minimum privileges level and your Astro-WISE user identity.

The minimum privileges level chosen for your Context defines from which privileges level on you want to have access to data objects. Thus, by lowering the minimum privileges level you add data objects to your data access scope. A minimum privileges level of \(i\) indicates data objects with privileges levels \(x \geq i\) will fall in your data access scope. For privileges levels \(\geq 3\) all data objects with that privileges level fall into your data scope. For privileges levels \(\leq 2\) a subset of the data objects with that privileges level belong to your data scope. Table 2 lists the rules which decide which data objects fall within your data access scope. Figure 1 shows a graphical representation of the resulting data access scope for each of the five minimum privileges levels that can be selected for your Context.

Table 2: This diagram illustrates the data access scope for User X who has selected PROJECT A in his Context. Together with the selected level from the 5 possible minimum privileges levels in Context it defines the data access scope. The diagram on top shows how sets of data objects are added to your data access scope as the minimum privileges level set in the Context is lowered. The table below it describes each of these sets which are defined as the data objects which have the values of the attributes _privileges, _project and _creator listed in the columns.

minimum privileges level of Context data access scope
5 5
4 5 + 4
3 5 + 4 + 3
2 5 + 4 + 3 + 2
1 5 + 4 + 3 + 2 + 1

where the data represented by 5, 4,… meet the following criteria for their attributes _privileges, _project and _creator:

dataset _privileges== _project== _creator==
5= 5 ANY project ANY creator
4= 4 ANY project ANY creator
3= 3 ANY project ANY creator
2= 2 PROJECT A ANY creator
1= 1 ANY project of which USER X is a member USER X

As an example, assume that you select your Context to have project=’BLUES’ and minimum privileges level=1. Then your data access scope will include

  • data with privileges==1 which are created by you in any project
  • data with privileges==2 which are in project=’BLUES’
  • data with privileges\(\geq\) 3 in all projects, also those of which you are not a member

Figure 1: Schematic representations of the data access scope for five different settings of Context (listed on top of each diagram). In all Context settings the Astro-WISE user is set to ‘User X’ and the project in Context to ‘Project A’. A separate diagram is shown for each of the five minimim privileges levels that User X can set for the Context. The columns on the diagram are divided over three projects. Per project there is one column for each project member. Such a column represents the data objects which belong to the project and which are created by that project member. The data objects can have 5 different privileges levels which are separated in rows. Thus each box in the diagram represents data objects which belong to the same project, have the same creator and the same privileges level (i.e., have the same values for attributes _project, _privileges and _creator). Green colored boxes indicate the data access scope of User X for the chosen Context settings: the data objects to which User X has access.

image5 image4 image3 image2 image1

Data creation scope

Upon creation/ingestion of an object the values of its attributes _project, _privileges and _creator are :

  • _project set to the id of the project of your Context
  • _privileges set to the minimum privileges level of your Context
  • _creator set to the id of your Astro-WISE user identity

Only project members can ingest/create data in a project. Data objects can not be created and ingested at all privileges levels. In private projects, the project members can create and ingest data objects only at privileges levels 1 and 2. Only project managers can then promote data objects to higher privileges levels up to privileges levels equal 5. This is called publishing and is described in the next section. In public projects the Astro-WISE users can ingest and create data object up to all privileges levels up to a maximum of 4.

Publishing of data objects

The philosophy of Astro-WISE is to share results across projects whenever beneficial. An example are shared calibration data. With time calibration scientists of projects will improve their knowledge and methods how to make the best calibration data for that instrument. For example, the improvements could be based on long term trend analysis. In this sense not individual nights but complete instruments become calibrated in Astro-WISE . The calibration scientists can share their improved calibration data with the Astro-WISE community by publishing his results to a privileges level of 3 or higher. Astro-WISE users, also in other projects, can then re-process their data using these improved calibration data. Furthermore, to improve their own calibrations of other periods they can inspect how the calibration was derived by the calibration scientist because they can access the data lineage of a data object. In conclusion, the idea is that with time data objects are promoted to higher privileges levels. This promotion to a higher privileges level is called publishing in Astro-WISE .

Only the creators and project managers can publish data objects of a project. Table 3 shows who can publish data objects to which privileges levels in public and private projects.

Table 3: This table lists to the privileges levels to which the creator and project manager of a project can publish data. The notation format is (possible start levels) \(\Rightarrow\) (possible end levels). The permitted levels are different for public and private projects.

project type creator project manager
private \((1) \Rightarrow (2)\) \((2,3,4) \Rightarrow (3,4,5)\)
public \((1,2,3) \Rightarrow (2,3,4)\) \((2,3,4) \Rightarrow (3,4,5)\)

So in private projects the project manager is solely responsible for the project data which is accessible to people outside the project team.

Publishing up to and including privileges level 3 is recursive: all dependencies are promoted to the published level as well.

Here is an example how to publish an object to privileges level 3:

awe> context.publish(object, privileges=3, commit=True, verbose=True)

The result of publishing is a larger number of people (Astro-WISE users and possibly persons without an Astro-WISE account) who can view and access the data as listed in Table 1.

Publishing of an object with dependencies in different projects

It is possible that an object has dependencies in different projects. If the object has privileges level \(\geq2\) it implies that the dependencies which belong to other projects have privileges level \(\geq3\). If the object has privileges level \(=1\) it can be the case that a dependency which belongs to another project has privileges level\(=1\). Publishing will fail in the latter case. The only way to publish MyDB data which has dependencies in other projects is to publish the dependencies first to privileges level 3 (or higher). This way the dependencies will always be visible to the parent object, independent of the project of the current Context. Now the parent object can be published to privileges level 2 (or higher).

Not everybody is authorized to publish. For the different privileges the following rules apply :

  • privileges 1 : only the creator is authorized to publish
  • privileges \(>=\) 2 (private projects): only the project manager is authorized to publish
  • privileges \(>=\) 2 (public projects): the project manager and the creator are authorized to publish

Unpublishing and invalidation

It is possible to demote the privileges level of a data object down to privileges level 2. This is called unpublishing. For example, unpublishing an object to privileges level 2 is done via:

awe> context.publish(object, privileges=2, commit=True, verbose=True)

Objects can be depublished to privileges level 2 if they are not referenced by data objects at privileges level 3 or higher. If they are, it might be desirable to invalidate the object. One can inspect whether a data object thisobject is valid as follows:

awe> thisobject.is_valid
1

1 means valid, 0 means invalid. To invalidate thisobject use:

awe> context.update_is_valid(thisobject, 0)

Deletion

Objects can be deleted from the database under the following restrictions. Every user can delete the data he created at the MYDB privileges level 1. For higher privileges levels only the project manager is allowed to delete. To delete a data object myobject from the database use:

awe> context.delete(myobject)

Objects can be deleted if they are not referenced by other data objects. If they are, it might be desirable to invalidate the object. One can inspect whether a data object thisobject is valid as follows:

awe> thisobject.is_valid
1

1 means valid, 0 means invalid. To invalidate thisobject use:

awe> context.update_is_valid(thisobject, 0)