HOW-TO Use your Astro-WISE Context to set data scopes¶
The Astro-WISE concept Context allows you to filter from the ocean of data objects in Astro-WISE the subset of data objects that you want to be currently visible to you and your processes. This is the ‘data access scope’ of your Context. At the same time your chosen Context also defines the logical subset to which the results of your processes will belong. This is the ‘data creation scope’ of your current Context. In other words, by configuring your Astro-WISE Context you define the logical subspaces in the ensemble of data objects in Astro-WISE in which you access data and in which you create data.
Astro-WISE Context¶
Your Astro-WISE Context is fully defined by choosing the settings of three Astro-WISE entities:
- Astro-WISE user identity
- project
- minimum privileges level
User¶
An Astro-WISE user is a person with an Astro-WISE database account. An Astro-WISE user has an id number and name (e.g., AWJJOPLIN). Each person has only one account. Thus, each person has a single identity within the Astro-WISE system. Each data object is associated with a single Astro-WISE user, the data object creator. This is the Astro-WISE user that created/ingested the data. Once established the creator of a data object cannot be changed.
At the awe-prompt the creator of data object myobject can be printed to the screen with
awe> myobject._creator # Returns user id
990
awe> from common.database.Database import database
awe> Database.users[myobject._creator] # Returns user name
'AWJJOPLIN'
Project¶
A project in Astro-WISE is a set of Astro-WISE users who collaborate on a common data set. A project has a project id, a name, a description, project members and optionally an instrument. One or more Astro-WISE users can be member of a project, and an Astro-WISE user can be member of more than one project as illustrated in Figure 1. Unlike an user each data object belongs to one and only one project. The project to which a data object belongs is chosen upon the creation/ingestion of the data entity and can not be changed after that.
Figure 1: This diagram shows which Astro-WISE users are project
members of Astro-WISE project A and B. A project in
Astro-WISE is a set of Astro-WISE users who collaborate on a
common data set. Project members are indicated in the diagram by a
green tick mark. An Astro-WISE user can be member of more than one
project. Each project has one or more Astro-WISE users as
project members. One or more of the project members can be
project manager (indicated with (pm)). A project manager has
additional abilities and responsibilities to manage project data.
At the awe-prompt the project to which data object myobject belongs can be printed to the screen with
awe> myobject._project #Returns project id
14
awe> from common.database.Database import database
awe> Database.projects[myobject._project] # Returns project name
'WFI@2.2m'
Some projects have all Astro-WISE users as members: these are called public projects. Some projects have a subset of Astro-WISE users as members: these are called private projects. One or more of the Astro-WISE users in a project can also act as project manager. A project manager has additional abilities and responsibilities to manage data objects that belong to a project. The extra abilities/responsibilities are for data objects at privileges levels of 3 and larger. At these privileges levels the data objects are visible to persons which are not project members. The concept of privileges is explained next.
An overview of all projects in the database, their members, the project’s manager etc. can be found on our webpages at the following address: http://process.astro-wise.org/Projects.
Privileges¶
Privileges of data objects determine which Astro-WISE users have access to the data object. Each data object in Astro-WISE has one of five privileges as listed in Table 1.
Table 1: Privilege levels of Astro-WISE
| privileges level | data is shared with |
|---|---|
| 1: MYDB | only the creator |
| 2: PROJECT | every member of the project to which the data object belongs |
| 3: ASTRO-WISE | all Astro-WISE users |
| 4: WORLD | the world: Astro-WISE users and persons without an Astro-WISE account |
| (latter via webservice DBViewer) | |
| 5: VO | again the whole world and data also available through the Virtual Observatory |
| via Virtual Observatory webservices |
The initial privileges of a data object are set upon the creation/ingestion of the data entity. They can be changed after that.
At the awe-prompt the privileges of data object myobject can be printed to the screen with
awe> myobject._privileges
1
Using Context¶
Your Astro-WISE Context is defined by your Astro-WISE user identity, the current project you choose and the current minimum privileges level that you choose. The Astro-WISE user that you are is defined upon login as you login with a certain user name (e.g. AWJJOPLIN). Upon login, also a current project and a current minimum privileges level are set. These two initial selections depend on your login configuration. At the Astro-WISE command-line prompt, the python class context is the interface to view and change the selection of project and privileges level. You can import context at the Astro-WISE prompt via:
awe> from common.database.Context import context
To see which minimum privileges level is currently active use
context.get_current_privileges()
awe> context.get_current_privileges()
1
To change the minimum privileges level call the
context.set_privileges(<privileges>) method:
awe> context.set_privileges(2)
To see which project is currently selected, use
context.get_current_project(). If the project object is printed, a
formatted overview will be given:
awe> print(context.get_current_project())
Project: SONATE (id = 14)
Description: 1700 Square degree 5 band survey of the equatorial strip and 2dF
South region
Instrument: OCAM
Maximum Privileges: 4 (WORLD)
Current Privileges: 1 (MyDB)
Instrument lists the instrument associated with the project. If it is None it means data from any instrument can be associated with the project. Maximum Privileges is the maximum privileges level which a project member can assign to a data object. To change to a project called ‘BLUES’ at the awe-prompt use:
awe> context.set_project('BLUES')
An user can only set projects of which the user is a member. When a project is selected without specifying the minimum privileges level they will be set to their default: 1.
To select at the same time the project and the minimum privileges level at the awe-prompt use:
awe> context.set_project('BLUES', privileges=2)
Data access scope¶
The philosophy in Astro-WISE is to increase efficiency in survey processing by sharing useful public data (such as calibration data). The philosophy of Context is therefore to allow users to select which data they want to see in their data access scope in addition to public data from Astro-WISE projects.
Whether a data object falls within the data access scope of the Context as currently configured depends on the values of 3 attributes which each data object has:
_privileges_project_creator
These attributes are compared to the corresponding 3 entities which configure your Context: the chosen values for project, minimum privileges level and your Astro-WISE user identity.
The minimum privileges level chosen for your Context defines from which
privileges level on you want to have access to data objects. Thus, by
lowering the minimum privileges level you add data objects to your data
access scope. A minimum privileges level of \(i\) indicates data
objects with privileges levels \(x \geq i\) will fall in your data
access scope. For privileges levels \(\geq 3\) all data objects with
that privileges level fall into your data scope. For privileges levels
\(\leq 2\) a subset of the data objects with that privileges level
belong to your data scope. Table 2 lists the rules
which decide which data objects fall within your data access scope.
Figure 1 shows a graphical representation of the
resulting data access scope for each of the five minimum privileges levels
that can be selected for your Context.
Table 2: This diagram illustrates the data access scope for User X who has
selected PROJECT A in his Context. Together with the selected level from the 5
possible minimum privileges levels in Context it defines the data
access scope. The diagram on top shows how sets of data objects are added to
your data access scope as the minimum privileges level set in the
Context is lowered. The table below it describes each of these sets which
are defined as the data objects which have the values of the attributes
_privileges, _project and _creator listed in the columns.
minimum privileges level of Context |
data access scope |
|---|---|
| 5 | 5 |
| 4 | 5 + 4 |
| 3 | 5 + 4 + 3 |
| 2 | 5 + 4 + 3 + 2 |
| 1 | 5 + 4 + 3 + 2 + 1 |
where the data represented by 5, 4,… meet the following criteria for their
attributes _privileges, _project and _creator:
| dataset | _privileges== |
_project== |
_creator== |
|---|---|---|---|
| 5= | 5 | ANY project | ANY creator |
| 4= | 4 | ANY project | ANY creator |
| 3= | 3 | ANY project | ANY creator |
| 2= | 2 | PROJECT A | ANY creator |
| 1= | 1 | ANY project of which USER X is a member |
USER X |
As an example, assume that you select your Context to have project=’BLUES’ and minimum privileges level=1. Then your data access scope will include
- data with privileges==1 which are created by you in any project
- data with privileges==2 which are in project=’BLUES’
- data with privileges\(\geq\) 3 in all projects, also those of which you are not a member
Data creation scope¶
Upon creation/ingestion of an object the values of its attributes
_project, _privileges and _creator are :
_projectset to the id of the project of your Context_privilegesset to the minimum privileges level of your Context_creatorset to the id of your Astro-WISE user identity
Only project members can ingest/create data in a project. Data objects can not be created and ingested at all privileges levels. In private projects, the project members can create and ingest data objects only at privileges levels 1 and 2. Only project managers can then promote data objects to higher privileges levels up to privileges levels equal 5. This is called publishing and is described in the next section. In public projects the Astro-WISE users can ingest and create data object up to all privileges levels up to a maximum of 4.
Publishing of data objects¶
The philosophy of Astro-WISE is to share results across projects whenever beneficial. An example are shared calibration data. With time calibration scientists of projects will improve their knowledge and methods how to make the best calibration data for that instrument. For example, the improvements could be based on long term trend analysis. In this sense not individual nights but complete instruments become calibrated in Astro-WISE . The calibration scientists can share their improved calibration data with the Astro-WISE community by publishing his results to a privileges level of 3 or higher. Astro-WISE users, also in other projects, can then re-process their data using these improved calibration data. Furthermore, to improve their own calibrations of other periods they can inspect how the calibration was derived by the calibration scientist because they can access the data lineage of a data object. In conclusion, the idea is that with time data objects are promoted to higher privileges levels. This promotion to a higher privileges level is called publishing in Astro-WISE .
Only the creators and project managers can publish data objects of a project. Table 3 shows who can publish data objects to which privileges levels in public and private projects.
Table 3: This table lists to the privileges levels to which the creator and project manager of a project can publish data. The notation format is (possible start levels) \(\Rightarrow\) (possible end levels). The permitted levels are different for public and private projects.
| project type | creator | project manager |
|---|---|---|
| private | \((1) \Rightarrow (2)\) | \((2,3,4) \Rightarrow (3,4,5)\) |
| public | \((1,2,3) \Rightarrow (2,3,4)\) | \((2,3,4) \Rightarrow (3,4,5)\) |
So in private projects the project manager is solely responsible for the project data which is accessible to people outside the project team.
Publishing up to and including privileges level 3 is recursive: all dependencies are promoted to the published level as well.
Here is an example how to publish an object to privileges level 3:
awe> context.publish(object, privileges=3, commit=True, verbose=True)
The result of publishing is a larger number of people (Astro-WISE users and possibly persons without an Astro-WISE account) who can view and access the data as listed in Table 1.
Publishing of an object with dependencies in different projects¶
It is possible that an object has dependencies in different projects. If the object has privileges level \(\geq2\) it implies that the dependencies which belong to other projects have privileges level \(\geq3\). If the object has privileges level \(=1\) it can be the case that a dependency which belongs to another project has privileges level\(=1\). Publishing will fail in the latter case. The only way to publish MyDB data which has dependencies in other projects is to publish the dependencies first to privileges level 3 (or higher). This way the dependencies will always be visible to the parent object, independent of the project of the current Context. Now the parent object can be published to privileges level 2 (or higher).
Not everybody is authorized to publish. For the different privileges the following rules apply :
- privileges 1 : only the creator is authorized to publish
- privileges \(>=\) 2 (private projects): only the project manager is authorized to publish
- privileges \(>=\) 2 (public projects): the project manager and the creator are authorized to publish
Unpublishing and invalidation¶
It is possible to demote the privileges level of a data object down to privileges level 2. This is called unpublishing. For example, unpublishing an object to privileges level 2 is done via:
awe> context.publish(object, privileges=2, commit=True, verbose=True)
Objects can be depublished to privileges level 2 if they are not referenced by data objects at privileges level 3 or higher. If they are, it might be desirable to invalidate the object. One can inspect whether a data object thisobject is valid as follows:
awe> thisobject.is_valid
1
1 means valid, 0 means invalid. To invalidate thisobject use:
awe> context.update_is_valid(thisobject, 0)
Deletion¶
Objects can be deleted from the database under the following restrictions. Every user can delete the data he created at the MYDB privileges level 1. For higher privileges levels only the project manager is allowed to delete. To delete a data object myobject from the database use:
awe> context.delete(myobject)
Objects can be deleted if they are not referenced by other data objects. If they are, it might be desirable to invalidate the object. One can inspect whether a data object thisobject is valid as follows:
awe> thisobject.is_valid
1
1 means valid, 0 means invalid. To invalidate thisobject use:
awe> context.update_is_valid(thisobject, 0)
