common.main package

Submodules

common.main.Config module

An interface to the canonical configurations and parameters of external programs

class common.main.Config.Config(filename=None, tmpbase='')

Bases: common.database.DBMain.DBObject

The base class for all persistent config classes.

The purpose of Config and derived classes is to provide a uniform interface for the canonical configuration of all external programs. This is a step towards process parameters, which would require proper classes for all external programs, a rather large feat to accomplish.

All KW_LISTS are now deprecated as all their information is now contained in the Config classes. Use an instance’s get_kw_list() method to return the original KW_LIST.

NOTE: _default_kw_list and _default_kw_dict are private class

attributes and should not be modified as they contain the default configuration of the class.

clone()

get_defaults()

Return a list of tuples (keyword, value)

get_kw_dict()

Return a dictionary of parameters and their values.

get_kw_list()

Return a list of (parameters, value) tuples.

is_config()

Test for being a configuration.

object_id

The object identifier

The object identifier is an attribute shared by all persistent instances. It is the prime key, by which object identity is established

program_name = ''

remove(filename=None)

Delete the settings file.

filename: name of file to delete

NOTE: If filename is given, it is used instead of self.filename.

If filename is None or if it doesn’t exist, nothing will be done.

save(filename=None)

Save the settings to a file.

filename: name of file to write settings to

NOTE: If filename is given, it is used instead of self.filename.

If neither are given, an error results.

set_defaults()

Set object attributes to their default values.

NOTE: Default values are defined in the Config module.

set_from_dict(config)

Set object attributes from a dictionary.

config: a dictionary containing the keywords as keys and values as

key values

set_from_keys(**config)

Set object attributes from keyword arguments (e.g., keyword0=value0, keyword1=value1, etc.).

set_from_list(config_list)

Set object attributes from a list of tuples.

config_list: a list of tuples (keyword, value)

show()

Show the settings as they will be written to the file.

update_dict(class_parameter_dict)

Method for setting class specific parameters and updating the class parameter dictionary.

class_parameter_dict: the class parameter dictionary, with class

specific parameter values

A parameter in the class_parameter_dict can be a plain value or a callable, which will be called for the value.

exception common.main.Config.ConfigError

Bases: Exception

class common.main.Config.NonPersistentConfig(filename=None, tmpbase='')

Bases: object

The base class for all non-persistent config classes.

The purpose of NonPersistentConfig and derived classes is to provide a uniform interface for the canonical configuration of all external programs that don’t need persistency in the database. This is a step towards process parameters, which would require proper classes for all external programs, a rather large feat to accomplish.

All KW_LISTS are now deprecated as all their information is now contained in the NonPersistentConfig classes. Use an instance’s get_kw_list() method to return the original KW_LIST.

NOTE: _default_kw_list and _default_kw_dict are private class

attributes and should not be modified as they contain the default configuration of the class.

clone()

get_defaults()

Return a list of tuples (keyword, value)

get_kw_dict()

Return a dictionary of parameters and their values.

get_kw_list()

Return a list of (parameters, value) tuples.

get_properties()

info()

Show information about the settings, specifically, the configurable parameters and their current settings.

is_config()

Test for being a configuration.

program_name = ''

remove(filename=None)

Delete the settings file.

filename: name of file to delete

NOTE: If filename is given, it is used instead of self.filename.

If filename is None or if it doesn’t exist, nothing will be done.

save(filename=None)

Save the settings to a file.

filename: name of file to write settings to

NOTE: If filename is given, it is used instead of self.filename.

If neither are given, an error results.

set_defaults()

Set object attributes to their default values.

NOTE: Default values are defined in the Config module.

set_from_dict(config)

Set object attributes from a dictionary.

config: a dictionary containing the keywords as keys and values as

key values

set_from_keys(**config)

Set object attributes from keyword arguments (e.g., keyword0=value0, keyword1=value1, etc.).

set_from_list(config_list)

Set object attributes from a list of tuples.

config_list: a list of tuples (keyword, value)

show()

Show the settings as they will be written to the file.

update_dict(class_parameter_dict)

Method for setting class specific parameters and updating the class parameter dictionary.

class_parameter_dict: the class parameter dictionary, with class

specific parameter values

A parameter in the class_parameter_dict can be a plain value or a callable, which will be called for the value.

exception common.main.Config.NonPersistentConfigError

Bases: Exception

class common.main.Config.Params(filename=None)

Bases: object

Base class for all non-persistent Params classes.

The purpose of Params and derived classes is to provide a uniform interface for the parameters of all external programs. This is a step towards process parameters, which would require proper classes for all external programs, a rather large feat to accomplish.

All PARAM_LISTS are now deprecated as all their information is now contained in the Params classes. Use an instance’s get_param_list() method to return the original PARAM_LIST.

get_defaults()

Return the default list of parameters.

get_param_list()

Return a list of parameters.

info()

Show information about the settings, specifically, the parameters.

is_config()

Test for being a configuration (Params are not configurations).

program_name = ''

remove(filename=None)

Delete the settings file.

filename: name of file to delete

NOTE: If filename is given, it is used instead of self.filename.

If neither are given, an error results.

save(filename=None)

Save the settings to a file.

filename: name of file to write settings to

NOTE: If filename is given, it is used instead of self.filename.

If neither are given, an error results.

set_defaults()

Set object attributes to their default values.

NOTE: Default values are defined in the Config module.

set_from_list(param_list)

Extend parameter list with param_list (duplicates are removed).

param_list: a list of parameters

show()

Show the settings as they will be written to the file.

update_list(param_list)

exception common.main.Config.ParamsError

Bases: Exception

common.main.Config.create_config(config_key, filename=None, class_defaults={}, tmpbase='')

Factory function for configurations.

All configurations should be created from here.

config_key: key corresponding to a particular Config class (see

config_dict)

filename: optional filename the configuration can be written to

class_defaults: dictionary of {parameter_name: parameter_value, …}

common.main.Config.create_params(params_key, filename=None, class_defaults=[])

Factory function for parameters.

All parameters should be created from here.

params_key: key corresponding to a particular Params class (see

params_dict)

filename: optional filename the parameters can be written to

class_defaults: list of [parameter_name, …]

common.main.InspectObject module

Persistent class to store files used for inspecting some result.

class common.main.InspectObject.InspectObject(pathname='', **kw)

Bases: common.database.DataObject.DataObject, common.database.DBMeta.DBMixin

A file storing inspection information.

creation_date

Date this object was created [None]

derive_timestamp()

Set the creation_date attribute.

filename

The name of the associated file [None]

globalname

The name used to store and retrieve file to and from Storage [None]

object_id

The object identifier

The object identifier is an attribute shared by all persistent instances. It is the prime key, by which object identity is established

purpose

Purpose of the inspection file [None]

exception common.main.InspectObject.InspectObjectError

Bases: Exception

Errors for InspectObjects.

common.main.OnTheFly module

########## Derived OnTheFly example ##########

# imports from common.main.OnTheFly import OnTheFly as OnTheFly_common,

OnTheFlyConfig as OnTheFlyConfig_common

from common.main.ProcessTarget import ProcessTarget

# define OnTheFlyConfig and OnTheFly class

class OnTheFlyConfig(OnTheFlyConfig_common) :

@classmethod def do_import(cls, cls_name_to_import):

# given the class_name return the class definition # This must be implemented for all classes that are on-the-fly processable #cls = … return cls

class OnTheFly(OnTheFly_common, ProcessTarget):

@classmethod def _exist(cls, attributes, config, object_id=None, parent=None, parent_attr=None):

# The exist method should for the given attributes return the object # from the database of class cls, if not present return None. # If the dependency is a list a list of objects should be returned # or an empty list. # The parent is the parent object requesting the object, the parent_attr # is the attribute of the parent that should be set # The config is the OnTheFlyConfig object # Always first call base class __init__ obj = OnTheFly_common._exist(attributes, config, parent=parent, parent_attr=parent_attr, **kwargs) if obj :

return obj

… rest of the implementation …

########## end Derived OnTheFly example ##########

WJ Vriend (wjvriend AT astro DOT rug DOT nl)

class common.main.OnTheFly.FixedDependencies(attributes=None, parent=None, config=None)

Bases: object

Class for handling fixed dependencies

Input is a dictionary called attributes;

the keys of this dictionary specify which dependency to fix the values specify the dependency to use

A dependency in the object tree is specified with :

class.attribute1.attribute2

class should be the requested class, attribute1 is an attribute of this class and attribute2 is an attribute of the class pointed by attribute1.

The dependency to use can be specified by class and object_id or class and custom identifiers.

An example :

# some persistent classes class A:

attr_b = persistent(‘pointer to class B’, B, None)

class B:

attr_c = persistent(‘pointer to class C’, C, None)

class C:

pass

# fix the dependency pointed by attr_b in class A to be the instance of class B with given object_id attributes = {

‘A.attr_b’ : (B, ‘1234567890abcdef’),

}

# fix the dependency pointed by attr_c of class B, which is pointed by attr_b of class A to be the instance of class C with given object_id attributes = {

‘A.attr_b.attr_c’ : (C, ‘1234567890abcdef’),

}

The class.attr1.attr2… chain can be as deep as needed but should at least be class.attribute.

The object_id can be a single value or a list (in case of a list dependency).

The object_id can also be None (or []) forcing not to use the dependency, setting it to None (or []).

Instead of object_id a custom identifier can be used, this must be a tuple. The translation between custom identifier and object must be implemented in the makeObjFromIds() method.

Example usage :

# using the attributes dict from the first example above fixed_dep = FixedDependencies(fixed_attrs_dict, parent=<instance of A>) obj, is_fixed = fixed_dep.getDependency(B, ‘attr_b’) # then obj will be the specified instance of B

# TODO add example of extendQuery()

extendQuery(cls, query)

Extend the given query with the fixed dependencies in the config cls is the class to query the dependencies on See the astro OnTheFly exist method for an implementetation example (TODO make common example)

extendQueryDeps(cls)

Return the dependencies for the ExtendQuery method

extendQueryGetObject(query)

get the object with the correct number of dependencies in the query

extendQueryNrOfDependencies(cls)

return the number of dependencies that will be fixed

getDependency(dep_class, dep_str)

If the tree index (cls.atrr.attr) is present in the fixed dependecy dict return specified object(s). Also return if the object was fixed. Fixed dependency dict layout :

{‘class.dep’ : (class, object_id) } {‘class.dep’ : (class, None) } {‘class.dep’ : (class, [object_id1, object_id2, …]) } {‘class.dep’ : (class, (custom identifier)) } {‘class.dep’ : (class, [(custom identifier1), (custom identifier2), …]) }

Fixed dependency when a dependency is a list, specify specific dependencies in the list:

{‘class.dep[object_id1].dep2’ : … } {‘class.dep[object_id2].dep2’ : … } {‘class.dep[custom identifier1].dep2’ : … } {‘class.dep[custom identifier2].dep2’ : … }

getDependencyObject(object_cls, object_id)

internal method to determine object from fixed object identifier

makeObjFromCustomIds(object_type, object_id)

From the object_id(s) make object(s) This method must be implemented in derived classes Return False to indicate to skip this object_id

makeObjFromIds(object_class, object_id)

If object_id is list, return list of objects Else return single object object_id is object_id or (datetime, chip, filter) tuple

max_contains = 20

message(msg)

Indent message with recusion level and call Message

classmethod obj_str(obj)

make string representation of given object

class common.main.OnTheFly.OnTheFly

Bases: object

Main methods :

get_onthefly(par) - Get the object onthefly, making all dependencies if needed - Is implemented by OnTheFly_make

uptodate(par) - Test if the object is up to date - Is implemented by OnTheFly_uptodate

exist(par) - Test if the object exists

All three methods have an interface version and internal version (starting with _)

Methods to be redefined by derived OnTheFly classes:

_exist(par)

Methods optional to be redefined :

onthefly_after_make onthefly_after_set_dependencies onthefly_after_uptodate onthefly_before_commit onthefly_before_make onthefly_init_attributes onthefly_processable onthefly_use_dependency

Other methods :

get_onthefly_dependencies check_dependency_for_mandatory

check_dependency_for_mandatory(dep_str, dep_new)

check if dependency is mandatory and present dep_str is the name of the attribute dep_new is the (attribute) object

classmethod exist(attributes=None, parent=None, parent_attr=None, config=None, object_id=None, **kwargs)

Interface method exist() for OnTHeFly Internally the method _exist is used This method initializes the config and does clean up

get_dependencies_flat()

Returns the instances that are used as dependencies of this object as a flat list. Objects that are used as a dependency twice will be returned only once. Dependencies that are not met are not returned. The attribute name(s) that correspond to the depencies are not returned, only the instances themselves.

classmethod get_onthefly(attributes, config=None, object_id=None, parent=None, parent_attr=None)

For the given attributes and class get the object, if the object does not exist -> make it. This is done recursively for all dependencies.

The object can only be made if there are raws

get_onthefly_dependencies(config=None)

Method retrieves a list of all dependencies used for on-the-fly processing Return: [[dependency, dependency class, dependency name], …]

is_flagged()

Is the object flagged valid (return False) or invalid (return True)

onthefly_after_make(config)

extra make statements, this method can be overridden by OnTheFly classes to implement extra make statements which will be executed after the make of self

onthefly_after_set_dependencies(config)

this method is called after the onthefly method set_onthefly_dependencies derived classes can implement this method to customize the dependencies setting after this has been done automatically by OnTheFly

onthefly_after_uptodate(dependencies_missing, dependencies_new, dependencies_obsolete)

The uptodate_object method determines if the object is uptodate, and will result in three listings :

dependencies_missing - missing mandatory dependencies dependencies_new - new dependencies dependencies_obsolete - current dependencies that are obsolete

Derived classes can override this method to tweak the listings

onthefly_before_commit(config)

This method is called after make and before committing the object Derived classes can override this method to change persistent properties just before they are committed to the database.

onthefly_before_make(config)

extra make statements, this method can be overridden by OnTheFly classes to implement extra make statements which will be executed before the make of self

classmethod onthefly_get_config(obj=None)

Return the common OnTheFlyConfig object, derived classes can return a specific Config object An optional object can be given to initialize the Config object

onthefly_init_attributes(attributes, dep_str, dependency, config)

hook for changing the attributes dict when going from self to a dependency in case of uptodate_dependencies(); when the dependency is a list, this method is called for each instance in the list in case of other; this method is called for each dependency, so in case of a list only for the whole list This could be implemented in derived classes, The default is to make a copy of the dictionary

classmethod onthefly_processable(attributes, config)

return True if class is OnTheFly processable, False if not derived classes can override

onthefly_use_dependency(property_obj, property_class, property_name)

Should the given dependency be OnTheFly processed ? Default True

retrieve_onthefly_dependencies(config=None)

Retrieves all the dependencies necessary to make the object.

uptodate(attributes, config=None)

This method checks if the object is up-to-date, for the given attributes. OnTheFly_uptodate class has the functionality for checking uptodate See OnTheFlyConfig for a description of config

walk_dependencies_backwards(done=None)

Walk the dependency tree of this object. See SCT.walk_tree_backwards

class common.main.OnTheFly.OnTheFlyConfig(cleanup=1, dependencies=None, derived=None, dpu_conn=None, fullscan=0, maxdepth=2, maxdepth_sci=-999, maxdepth_cal=-999, parameters=None, testrun=0, uptodate_parameters=True, verbose=0)

Bases: object

The configuration class for OnTheFly processing.

Configuration can be done by attributes or redefinition of methods.

Attributes :

• cleanup 0 | 1 cleanup onthefly values after get_onthefly or uptodate call
• dependencies dict
• derived dict
• fullscan 0 | 1 make a full scan of the object (all the dependencies to maxdepth)
• maxdepth 0…N maximum recursion depth into calibration tree, do science full
• maxdepth_sci 0…N maximum science recursion depth
• maxdepth_cal 0…N maximum calibration recursion depth
• parameters dict
• testrun 0 | 1 toggle a testrun

• uptodate_parameters

  0 | 1

• verbose 0 | 1 toggle verbose

The depth specifies how deep to iterate the class tree: -1 indefinite 1 only check if the object is present 2 check if the object uses the latest dependencies 3 check if the dependencies use the latest dependencies X etc … The science and cal depth specify how deep to iterate science and calibration object trees

The parameters dictionary has the generic form:

{‘parameter.specifier’: parameter_value, ….}

So for example to specify the process parameter MAXIMUM_PSF_DIFFERENCE and the swarp configuration BACK_FILTERSIZE for a RegriddedFrame :

{‘RegriddedFrame.process_params.MAXIMUM_PSF_DIFFERENCE’: 0.35,

‘RegriddedFrame.swarpconf.BACK_FILTERSIZE’: 5}

FixedDependenciesHandler

alias of FixedDependencies

add_to_cache(result, cls, attributes)

add the resulting object to the cache

classmethod after_uptodate_parameters(obj, parameters_differ)

method is called after uptodate_parameters is done

before_set_onthefly_parameters(obj)

method is called before the onthefly process parameters are set

change_recursion_level(cls, parent, change)

change the recursion level(s) with change

classmethod check_attributes(attributes, target_class)

check or change the attributes dict this could be implemented in derived classes

check_dict(d)

check that the keys in the dictionary all start with the same class

check_level(cls, parent=None)

Method checks if maximum level depth has been reached

circular_recursion(parent_class, dep_class)

Detect a circular (endless) recursion loop return True is circular recursion is detected and skip the current attribute Default False

clearOntheflyAttributes()

clear all the set _ONTHEFLY_VALUE and _ONTHEFLY_DICT attributes

dependencies

depth_magic_value = -999

classmethod do_import(cls_name_to_import)

Import the cls_name as class_value This method must be redefined !!

getFixedDependencies(attributes, dep_class, dep_str, parent=None)

return the fixed dependencies from the fixed dependency handler

getFixedDependenciesHandler(attributes, parent=None)

return an instant of the fixed dependency handler

getTreeIndex()

return the tree index, that is the index defining the point in the tree; class_a.attr_b.attr_c…

classmethod get_cache_key(object_class, attributes)

In test mode query results are cached, given the attributes return the key to cache the object with the given attributes

get_dependencies()

get_from_cache(cls, attributes)

if an object was cached for the current request, return it

get_parameters()

higher_recursion_level(cls, parent)

higher the recursion level(s)

initForClass(cls)

initialize self for the given class raise depth of corresponding category

initInternals()

initialize internal attributes

level_change(current_class, parent)

Should we change the level depth, default True

lower_recursion_level(cls, parent)

lower the recursion level(s)

message(msg)

Indent message with recusion level and call Message

classmethod obj_str(obj)

make string representation of given object

parameters

recurse_to_dependencies(a_object, dep_class, attribute_name)

recurse towards a dependency: - raise the recursion level - grow the tree index

recurse_to_parent(a_object, dep_class)

recurse towards the parent: - lower the recursion level - shrink the tree index

retrieve(obj)

Retrieve the file associated with the object

set_attributes(obj, attributes)

Set attributes on self from attributes dict only used in test mode for GUI

set_dependencies(deps)

set_parameters(pars)

store(obj)

Store the file associated with the object

uptodate_skip(parent, dependency)

Return True if the uptodate check can be skipped for the given dependency, default False

use_cache(cls, attributes)

Should the cache be used (in _exist) for the given class and attributes

exception common.main.OnTheFly.OnTheFlyError

Bases: Exception

class common.main.OnTheFly.OnTheFly_base

Bases: object

Base class for OnTheFly_make and OnTheFly_uptodate TODO * move common methods here * make common __init_ method * implement dependency iterator

get_fixed_parameters()

return the parameter type and dictionaries for the current object

message(msg)

Indent message with recusion level and call Message

classmethod message_format(msg, depth=0, config=None)

Indent message with recusion level

classmethod obj_str(obj)

make string representation of given object

classmethod set_onthefly_value(objects, value, config=None, force=False, value_dict=None)

Set the onthefly value on the object or list of objects Only if obj or list of objects do not have it set yet or force is set Also keep track of the depth (in the class tree) where the value is set

class common.main.OnTheFly.OnTheFly_make(cls, attributes, object_id, config, parent=None, parent_attr=None)

Bases: common.main.OnTheFly.OnTheFly_base

class for onthefly initializing process targets the attributes define which object to ‘get’ the config object contains configuration

First is checked if the targets exists, which is checked for being uptodate, if not existing or not uptodate the target is (re)made

get()

get the target

get_recursive(object_id, target_class, parent, parent_attr)

get the target

onthefly_do_make(new_object)

Retrieve all dependencies and then make, store and commit the object

onthefly_make(target_class)

Set the dependencies of the object, set the parameters and call the make method of the requested class

set_onthefly_dependencies(new_object)

Determine every dependency and set it to new_object !!!! for internal use only !!!!

set_onthefly_parameters(obj)

set the (fixed) process parameters

class common.main.OnTheFly.OnTheFly_uptodate(a_object, attributes, config)

Bases: common.main.OnTheFly.OnTheFly_base

class for testing if an object is uptodate initialize with object to test and - attributes the attributes to test the object against - config the configuration, see OnTheFlyConfig

test()

test object for being uptodate, this done recursively for all onthefly dependencies

uptodate_dependencies(a_object, uptodate_dict)

method for internal use call the uptodate method of all dependencies

uptodate_object(a_object, uptodate_dict)

method for internal use checks if object uses the newest version all its dependencies

uptodate_parameters(a_object, uptodate_dict)

check if any parameters are different then parameters specifed

uptodate_recursive(a_object)

Test object for being uptodate The uptodate check consists of three parts 1. check the process parameters 2. Is a flag set 3. Are there newer versions for any dependency 4. Call uptodate of all dependencies

common.main.ProcessTarget module

The common ProcessTarget

This defines a processable object.

class common.main.ProcessTarget.ProcessTarget

Bases: object

The common ProcessTarget defines a make-able object.

See the ProcessTarget in astro for a specific example

PROCESS_TIME = 0

check_mandatory_dependencies()

check if all the mandatory dependencies are set

compare(*args, **kwargs)

Quality Control – basic trend analysis

The compare() method is used for default trend analysis. This is done by comparing with a previous version of the same object (last weeks bias, for example). This may be as simple as comparing attribute values, but may also involve more complex computations (e.g., subtracting the two images, and analysing the residuals)

get_dependencies()

Return a list of names of attributes that are required for make()

inspect(*args, **kwargs)

Quality Control – visual inspection

Visual inspection of the data remains a powerful tool in quality control. The inspect() method provides the mechanism to record the results of visual inspection for posterity.

classmethod is_cal()

Test for being cal, derived raw classes should set _IS_CAL to 1

classmethod is_config()

Test for being config, derived raw classes should set _IS_CONFIG to 1

classmethod is_raw()

Test for being raw, derived raw classes should set _IS_RAW to 1

classmethod is_science()

Test for being science, derived raw classes should set _IS_SCIENCE to 1

classmethod is_seq()

Test for being seq, derived raw classes should set _IS_SEQ to 1

classmethod is_support()

Test for being config, derived raw classes should set _IS_SUPPORT to 1

make()

Construct this object from the provided data.

The make method is expected to produce all data and compute all values associated with this object (i.e.: reduce the data). It is assumed that the object will have a number of dependencies which have to be satisfied. A dependency is assumed to be satisfied if the dependency is not None. In the case that the dependency is itself a ProcessTarget, the dependency should be made.

Use the method get_dependencies() to get a list of attributes on whose values this target depends.

mandatory_dependencies = ()

verify(*args, **kwargs)

Quality Control – internal consistency

This method inspects the values of various attributes of the object to see if these are within the expected range for that object. The purpose of this method is mostly to perform sanity-checks on measured results. It is assumed that the required measurements (for example image statistics) are done during data reduction (i.e. while executing make()), and stored in persistent attributes.

exception common.main.ProcessTarget.ProcessTargetError

Bases: Exception

Module contents