astro.services.process2 package

Submodules

astro.services.process2.HtmlProcess module

File HtmlProcess.py Author WJ Vriend (wjvriend[AT]astro.rug.nl) Date Apr 2005

class astro.services.process2.HtmlProcess.HtmlProcess(args)

Bases: common.services.general.HtmlBase.HtmlBase

The main html generation class of the target processor

see Process.py for helper methods like get_url()

ERRORS_START = '<p style="font-weight:bold;color:red;">Error(s) encountered <br>%s</p>'
author_email = 'wjvriend@astro.rug.nl'
author_name = 'Willem-Jan Vriend'
getCookies()

set project and instrument in cookie, also pickle the preferences only send cookie when : - sendCookie == True (HtmlProcessPreferences is submitted) - project and/or instrument is changed

getHtml()

get the class to generate html

get_generator_input()

get the html generator for the input

get_generator_option()

get the html generator for the options screen

get_generator_result()

get the html generator for the resuts query/process) screen

get_generator_targets()

get the html generator for the result

get_generator_type()

get the html generator for the type selecting

get_project_instrument()

determine the instrument and project

htmlStatusMsg()

make html from the set status messages

init_master_dict()

The central place for storing data between sessions is the master_dict this is stored as hidden input var on the web page or is generated if linked from external service

master_dict_from_external(external_service)

this methods translates the arguments given by the external service to the internally used master_dict

no_project_only_projects = ('KIDS',)
setStatusMsg(msg, to_log=True)

set error status message and make other vars empty

setTemplateVars()

set the template vars

supported_instruments = ('ACAM', 'DECAM', 'LOFAR', 'MEGACAM', 'MDM8K', 'OCAM', 'OMEGACAM', 'SUP', 'VIRCAM', 'WFI', 'WFC', 'WFCAM', 'WSRT')
class astro.services.process2.HtmlProcess.HtmlProcessIntro(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

getHtml()

generate html for the intro screen

class astro.services.process2.HtmlProcess.HtmlProcessType(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

classmethod arg_handler(parent)

get the object type and place it in the master_dict

getHtml()

generate html

classmethod is_supported(cls_name)

is the class with cls_name supported by the target processor ?

opt_cal_objects = [['AstrometricParameters', 'Astrometric', 1], ['ColdPixelMap', 'ColdPixel', 1], ['DarkCurrent', 'DarkCurrent', 1], ['DomeFlatFrame', 'DomeFlat', 1], ['FringeFrame', 'Fringe', 1], ['GainLinearity', 'Gain', 1], ['GAstrometric', 'GAS', -1], ['HotPixelMap', 'HotPixel', 1], ['IlluminationCorrectionFrame', 'IllumCorrFrame', -1], ['IlluminationCorrection', 'IllumCor', -1], ['BiasFrame', 'MasterBias', 1], ['MasterFlatFrame', 'MasterFlat', 1], ['NightSkyFlatFrame', 'NightSkyFlat', -1], ['PhotometricParameters', 'Photometric', 1], ['PhotSrcCatalog', 'PhotSrcCat', 1], ['ReadNoise', 'ReadNoise', 1], ['TwilightFlatFrame', 'TwilightFlat', 1]]
opt_img_objects = [['ReducedScienceFrame', 'ReducedScience', 1], ['RegriddedFrame', 'RegriddedFrame', 1], ['CoaddedRegriddedFrame', 'CoAddedFrame', 1], ['SourceList', 'SourceList', 1], ['AssociateList', 'AssociateList', 1], ['GalPhotModel', 'GalPhotModel', 1], ['GalFitModel', 'GalFitModel', 1]]
opt_text = 'Select the target for processing. On the left are the image pipeline targets, on the right the calibration.'
rad_cal_objects = []
rad_img_objects = [['SourceList', 'SourceList', 1]]
rad_text = 'Select the target for processing.'
astro.services.process2.HtmlProcess.fill_master_dict(master_dict=None)

initialize the master dict, empty or (partial) filled

astro.services.process2.HtmlProcessCodefile module

File HtmlProcessCodefile.py Author WJ Vriend (wjvriend@astro.rug.nl)

class astro.services.process2.HtmlProcessCodefile.HtmlProcessCodefile(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

MAX_SIZE = 20000000
MAX_SIZE_REACHED = 'uploaded file size (%d) is larger then maximum (%d)'
WARNING = '\n<script type="text/javascript">\nfunction useCodefile()\n{\n // disable clear input filename when selecting codefile\n document.getElementById(\'codefile\').value = \'\';\n document.Process.submit();\n}\n</script>\n\n<p>Chose wich zipped codefile to use. The \'default\' codefile is based on the source code from cvs. The other\ncodefile(s) are uploaded by the user.</p>\n<table class="ProcessTable ZebraTableBlue">\n<tr><th><th>User<th>Filename<th>Creation<th>Size\n<tr><td><input type="radio" name="select_codefile" value="1" %s>\n <td>default<td>%s<td>%s<td>%d\n<tr><td><input type="radio" name="select_codefile" value="2" %s>\n <td>%s<td>%s<td>%s<td>%d\n</table>\n<p><input type="button" value="Submit" onclick="useCodefile();"></p>\n<hr>\n<p style="color:red;">You can upload a new codefile but then the existing file will be overwritten!</p>'
classmethod arg_handler(parent)

codefile is uploaded

getHtml()

astro.services.process2.HtmlProcessGalFit module

File HtmlProcessGalFit.py Author WJ Vriend (wjvriend[AT]astro.rug.nl) Class to render html pages for GalFitModels in the Target Processor

class astro.services.process2.HtmlProcessGalFit.HtmlProcessGalFit(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

classmethod arg_handler(parent)

handle the queries set by the user

do_component_page()

generate the GalFit start page, for selecting components

do_constraint_page()

generate the GalFit constraint page, for selecting constraints

do_overview_page()

generate overview page, sohwing the components and constraints

galfit_components = ['GalFitDevauc', 'GalFitExpdisk', 'GalFitGaussian', 'GalFitMoffat', 'GalFitKing', 'GalFitNuker', 'GalFitSersic', 'GalFitSky']
galfit_constraints = ['GalFitRelConstraint', 'GalFitAbsConstraint', 'GalFitDiffConstraint', 'GalFitRatioConstraint']
getHtml()

generate html

classmethod handle_components(parent)

the user has selected components and input values

classmethod handle_constraints(parent)

the user has selected constraints

classmethod handle_overview(parent)

the user has submitted the overview, make the target and go to process submit page

skip_attrs = ['Z', 'name']
store_vars = ['components', 'components_input', 'components_fixed', 'constraints', 'constraints_input', 'components_set', 'constraints_set']
astro.services.process2.HtmlProcessGalFit.get_galfit_class(cls_str, module)

import the class and return it

astro.services.process2.HtmlProcessGalPhot module

File HtmlProcessGalPhot.py Author WJ Vriend (wjvriend[AT]astro.rug.nl) Class to render html pages for GalPhotModels in the Target Processor

class astro.services.process2.HtmlProcessGalPhot.HtmlProcessGalPhot(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

classmethod arg_handler(parent)

handle the queries set by the user

getHtml()

generate html

astro.services.process2.HtmlProcessInput module

HtmlProcessInput class definition

Author WJ Vriend (wjvriend[AT]astro.rug.nl)

class astro.services.process2.HtmlProcessInput.HtmlProcessInput(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

Base class for HtmlProcessInputGraph and HtmlProcessInputParameters with shared functionality

choseAstrometry()

For some classes the user should chose which astrometry to use; local vs global

determineRaw()

Determine the raw class string

getInputTypes()

Get the possible input classes

astro.services.process2.HtmlProcessInputGraph module

File HtmlProcessGraph.py Author WJ Vriend (wjvriend[AT]astro.rug.nl)

Class for generating html code for the target processor to graphically select input parameters

class astro.services.process2.HtmlProcessInputGraph.HtmlProcessInputGraph(parent, args)

Bases: astro.services.process2.HtmlProcessInput.HtmlProcessInput

CLRS_FILTER = ['FFFF00', 'FFCC00', 'FF9900', 'FF6600', 'FF3300', 'FF0000', '00FFFF', '00CCFF', '0099FF', '0066FF', '0033FF', '0000FF', 'FF00FF', 'FF00CC', 'FF0099', 'FF0066', 'FF0033', 'FF0000']
CLRS_INDEX = [0, 6, 12, 2, 8, 14, 4, 10, 16, 1, 7, 13, 3, 9, 15, 5, 11, 17]
CLR_BAR = '4444EE'
CLR_BG_GREY = 'E0E0E0'
CLR_WHITE = 'FFFFFF'
DARKENING = 0.65
HTML_ERROR = '<html><body><h1>Error</h1><h2>%s</h2></body></html>'
NONE = '<none>'
classmethod arg_handler(parent)

handle the given arguments

bar_height = 20
bars_height = 100
bars_left = 10
bars_top = 10
bars_total_width = 800
determine_counts(rows)

determine the raw and processed counts from the given rows

drawBackground(tmpl)

draw alternating background color bars, depending on the timescale

drawBackgroundCounts(tmpl, counts)

draw alternating background color bars using counts for the (relative) width

generateYearSelect()

select all distinct years from DATE_OBS

getHtml()

generate html

getLeftWidthBottomHeight(night)

determine the coordinates (left, width, bottom, height) of the bar for night

get_dt_start_end(night)

get the start and end of the night

get_key_from_row(row)

return key from given row

get_night_dict(row, dt_start, dt_end, ra=None, dec=None)

get a dictionary with the values specifying the night

group_by_default = 'Filter'
group_by_per_target = {'BiasFrame': 'Template', 'DarkCurrent': 'Template', 'GainLinearity': 'Template', 'HotPixelMap': 'Template', 'ReadNoise': 'Template'}
initLegendCounts(group_attribute)

determine the distinct filter names or template ids from the given recordset and couple them with colors in self.legend_colors

init_data()

Retrieve data from the database and initialize raw_counts, processed_counts and date_elements So we can loop through the data in a structured manner

is_valid_row(row)

check if row is valid

legend_top = 160
legend_x = 550
make_tooltip(night, rows, raw_count, processed_count)

make a tooltip table with info for the night, filter/template/obs_block

ob_enabled_targets = ('ReducedScienceFrame', 'AstrometricParameters', 'RegriddedFrame', 'CoaddedRegriddedFrame', 'BiasFrame', 'DarkCurrent', 'DomeFlatFrame', 'GainLinearity', 'TwilightFlatFrame', 'ReadNoise')
plotBar(tmpl, top, left, height, width, color, background, z_order, other, cursor)

plot a bar / block in the graph

plotBars(tmpl)

plot the bars; representing a distinct filter/night (and grid_target for coadd)

plotGraph(tmpl)

set the graph text

plotNightBars(tmpl, night, row_dict, cursor)

plot a bar for the specific night / filter / type combination, stating the number of raws, processed, date obs and filter in a tooltip

plotOutline(tmpl)

plot outlines of the graph

setExtraOptional(tmpl)

return a html combo box for input type selection

set_all_empty(tmpl)

set emtpy values

set_graph_empty(tmpl)

set emtpy values for the graph text

set_legend_colors(tmpl)

plot the legend; which color represents which filter/template and counts

set_legend_empty(tmpl)

set emtpy values for the legend

store_vars = ['Year', 'Quarter', 'Month', 'Week', 'filter', 'hide_flagged', 'project_only', 'group_by', 'input_type', 'chip', 'object_name', 'ra', 'dec', 'p_m', 'astrometry']
template_enabled_targets = ('ReducedScienceFrame', 'AstrometricParameters', 'RegriddedFrame', 'CoaddedRegriddedFrame', 'BiasFrame', 'DarkCurrent', 'DomeFlatFrame', 'GainLinearity', 'HotPixelMap', 'TwilightFlatFrame', 'ReadNoise')
astro.services.process2.HtmlProcessInputGraph.getNoonCorrection(instrument)

determine noon of the instrument in ut, for use in oracle queries

astro.services.process2.HtmlProcessInputGraph.is_valid_OB(observing_block)

check if observing_block is valid (>0 or -1)

astro.services.process2.HtmlProcessInputGraphDb module

HtmlProcessInputGraphDb.py

Database layer for HtmlProcessInputGraph

WJ Vriend

class astro.services.process2.HtmlProcessInputGraphDb.DbProcessInputGraph(raw, target, group_by, html_obj)

Bases: object

class generates sql and returns record sets for HtmlProcessInputGraph

I_COUNT = 1
I_DATE_OBS = 0
I_DEC = -999
I_RA = -999
NO_FILTER = 'no filter'
NO_OB = 'no OB'
NO_TEMPLATE = 'no template'
attr_start = 2
filter_attrs = ('name', 'mag_id', 'has_fringes')
filter_key = 2
filter_order_by = 'name'
getProcessedRS(raw_rs)

get the processed recordset

getRawRS()

get a recordset with raw objects

classmethod get_attr(row, index)

return the attr at index in the row

get_coord(row, field_centre=True)

return the coordinates, if selected

classmethod get_count(row)

return the attr at index in the row

classmethod get_date_obs(row)

return the attr at index in the row

classmethod get_night(row)

return the attr at index in the row

ob_attrs = ('id', 'dictionary_id', 'program_id', 'start', 'name', 'group', 'PI_COI_id', 'PI_COI_name')
ob_order_by = 'id'
observing_block_key = 2
template_attrs = ('id', 'dictionary_id', 'name', 'exposures', 'sequencer', 'start', 'obs_id', 'obs_start', 'index', 'version', 'strategy', 'category', 'technique', 'type')
template_key = 2
template_order_by = 'name'
class astro.services.process2.HtmlProcessInputGraphDb.SqlBase(attr_names, html_obj, dependency_name, order_by)

Bases: object

base class for the SQL generators

addWhereFlags(where, where_args, sql_attr, cls, ignore_quality_flags=False)

extend the sql where with the possible flags

generate()
getWhere(table_1=None, table_2=None, class_1=None, class_2=None, forced_date_alias=None)

Make a default where statement using date_obs, filter and instrument 1 is the raw table or class 2 is the processed table or class where_args could be a list of where arguments

sql_coadd(select, ra, dec, table, from_extra, where, where_outer, orderby)

SQL template for coadds

sql_date_obs(select, sql_from, where, where_outer, orderby, date_alias='t', date_obs='DATE_OBS')
class astro.services.process2.HtmlProcessInputGraphDb.SqlBaseProcessed(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBase

class astro.services.process2.HtmlProcessInputGraphDb.SqlBaseRaw(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBase

class astro.services.process2.HtmlProcessInputGraphDb.SqlProcessed(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseProcessed

generate()

get the sql to query processed objects with multiple raws as input

class astro.services.process2.HtmlProcessInputGraphDb.SqlProcessedAP(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseProcessed

SQL generator for processed AstrometricParameter’s (AP)

generate()

generate SQL

class astro.services.process2.HtmlProcessInputGraphDb.SqlProcessedCoadd(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseProcessed

generate()

get the sql for processed coadd’s, or regrids in coadds

class astro.services.process2.HtmlProcessInputGraphDb.SqlProcessedMasterFlat(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseProcessed

generate()

get the sql for processed MasterFlatFrames, the Twilights, Domes and Nightsky’s which have processed in a MasterFlatFrame

class astro.services.process2.HtmlProcessInputGraphDb.SqlProcessedPhotSrcCat(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseProcessed

generate()

get the sql to query processed PhotSrcCatalog objects

class astro.services.process2.HtmlProcessInputGraphDb.SqlProcessedPhotometric(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseProcessed

generate()

return the sql for querying processed PhotometricParameters objects

class astro.services.process2.HtmlProcessInputGraphDb.SqlProcessedPixelMap(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseProcessed

generate()

get the sql to query processed pixelmap objects

class astro.services.process2.HtmlProcessInputGraphDb.SqlProcessedRegrid(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseProcessed

SQL generator for processed Regrid’s and Reduced’s

generate()

generate SQL

class astro.services.process2.HtmlProcessInputGraphDb.SqlProcessedSourceList(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseProcessed

generate()

get the sql for processed SourceList’s, (coadds in sourcelist) this is a specialized version for SourceList

class astro.services.process2.HtmlProcessInputGraphDb.SqlRaw(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseRaw

sql generator for most raw classes

generate()

generate SQL to query the possible raw input

class astro.services.process2.HtmlProcessInputGraphDb.SqlRawCoadd(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseRaw

generate()

get the raw recordset for a coadd, get the regrids with date_obs in selected period and pixel_scale is default

class astro.services.process2.HtmlProcessInputGraphDb.SqlRawMasterFlat(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseRaw

generate()

get the raw record set for a masterflat frame, combine twilight, dome and nightsky recordsets

class astro.services.process2.HtmlProcessInputGraphDb.SqlRawPhotometric(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseRaw

generate()

construct the sql to query for raw input for PhotometricParameter

class astro.services.process2.HtmlProcessInputGraphDb.SqlRawPixelMap(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseRaw

generate()

get sql for querying raw pixelmap objects

class astro.services.process2.HtmlProcessInputGraphDb.SqlRawSourceList(attr_names, html_obj, dependency_name, order_by)

Bases: astro.services.process2.HtmlProcessInputGraphDb.SqlBaseRaw

generate()

get sql for querying raw objects, using date_obs, specialized version for SourceList’s

astro.services.process2.HtmlProcessInputGraphDb.getGastromWhere(attr, astrometry)

Return the Gastrom where clause, based on the astrometry setting in the parent (local, global or any)

astro.services.process2.HtmlProcessInputGraphDb.get_onthefly_template(target_class, target, raw, ignore_readnoise_template)

use the getTemplate method of OnTheFly to get the template name, it needs the depency string, which is not known here, so hardcode (TODO how to use getTemplate in a more transparent way ?) -> implement the dependency name in getTempate, set it when no dependency name is given

astro.services.process2.HtmlProcessInputParameters module

File HtmlProcessInputParameters.py Author WJ Vriend (wjvriend[AT]astro.rug.nl) Date Jul 2007

class astro.services.process2.HtmlProcessInputParameters.HtmlProcessInputParameters(parent, args)

Bases: astro.services.process2.HtmlProcessInput.HtmlProcessInput

DATE_NONE = '0000-00-00'
DEFAULT_P_M = '0.5'
TIME_NONE = '00:00:00'
classmethod arg_handler(parent)

handle the given arguments

date_format_python = '%Y-%m-%d'
getHtml()

generate html

classmethod init_date(parent, parent_msg, date_str, field_name)

init a date object from the date_str

classmethod init_time(parent, parent_msg, time_str, field_name)

init a time object from the time_str

initializeTemplate(tmpl)

initialize the template, filling vars known before processing

classmethod initialize_dates(parent, date_str, time_str, date_2_str, time_2_str, parent_msg=True)

if only date is given initialize self.date if date and time are given initialize self.datetime if date_2 (and time_2) are given use them for date_max

store_vars = ['filter', 'chip', 'date_str', 'time_str', 'date_2_str', 'time_2_str', 'object_name', 'ra', 'dec', 'p_m', 'hide_flagged', 'input_type', 'obs_block', 'template', 'astrometry']
time_format_python = '%H:%M:%S'

astro.services.process2.HtmlProcessJobStatus module

File HtmlProcessJobStatus.py Author WJ Vriend (wjvriend@astro.rug.nl)

Module with astro class showing for showing a single job.

class astro.services.process2.HtmlProcessJobStatus.HtmlProcessJobStatus(parent, args)

Bases: common.services.process.HtmlProcessJobStatus.HtmlProcessJobStatus

(overridden) return preview or image link

IDS = ['ALID', 'GPID', 'GPLID', 'SLID']

(overriden) return link to DbView for this object

getExtraTableData(obj)

(overriden) define extra elements for the object overview table

getExtraTableHeading()

(overriden) define extra heading elements for the object overview table

getLiExtra()

(override) return extra <li> messages, shown on top of the the page

(overridden) return link to quality server

astro.services.process2.HtmlProcessJobs module

File HtmlProcessJobs.py Author WJ Vriend (wjvriend@astro.rug.nl)

class astro.services.process2.HtmlProcessJobs.HtmlProcessJobs(parent, args)

Bases: common.services.process.HtmlProcessJobs.HtmlProcessJobs

astro specific version for showing individual jobs

getExtraJavaScript()

(overridden) return extra javascript code to include

showJob()

(overridden) show the details of a job

showJobs()

(overridden) generate the html to show the job overview append hidden input vars for job control

astro.services.process2.HtmlProcessNameResolver module

Target Processor Module html form resolving names to coordinates

class astro.services.process2.HtmlProcessNameResolver.HtmlProcessNameResolver(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

classmethod arg_handler(parent)

handle the given arguments

getHtml()

generate html

astro.services.process2.HtmlProcessObject module

author : WJ Vriend (wjvriend@astro.rug.nl) Module for generating html pages with OBJECT# names

class astro.services.process2.HtmlProcessObject.HtmlProcessObject(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

classmethod arg_handler(parent)

handle the given arguments

default_letter = 'A'
default_type = ['RawScienceFrame', <class 'astro.main.RawFrame.RawScienceFrame'>]
getHtml()

generate html

letters = '1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ'
supported_types = {'AssociateList': ['SourceList', <class 'astro.main.SourceList.SourceList'>], 'BiasFrame': ['RawBiasFrame', <class 'astro.main.RawFrame.RawBiasFrame'>], 'CoaddedRegriddedFrame': ['RawScienceFrame', <class 'astro.main.RawFrame.RawScienceFrame'>], 'DomeFlatFrame': ['RawDomeFlatFrame', <class 'astro.main.RawFrame.RawDomeFlatFrame'>], 'ReadNoise': ['RawBiasFrame', <class 'astro.main.RawFrame.RawBiasFrame'>], 'ReducedScienceFrame': ['RawScienceFrame', <class 'astro.main.RawFrame.RawScienceFrame'>], 'RegriddedFrame': ['RawScienceFrame', <class 'astro.main.RawFrame.RawScienceFrame'>], 'SourceList': ['RawScienceFrame', <class 'astro.main.RawFrame.RawScienceFrame'>], 'TwilightFlatFrame': ['RawTwilightFlatFrame', <class 'astro.main.RawFrame.RawTwilightFlatFrame'>]}

astro.services.process2.HtmlProcessParameters module

File HtmlProcessParameters.py Author WJ Vriend (wjvriend@astro.rug.nl)

class astro.services.process2.HtmlProcessParameters.HtmlProcessParameters(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

ABS_MAX_RECURSION_DEPTH = 20
classmethod arg_handler(parent)

form is submitted get process parameters from the arguments

convertTreeElem(obj_instance, obj_default, tree_str)

make html li items for every parameter set of the given object

doTemplateBlock(keys, config_fixed, config_instance, config_default, config_instr_specific, tree_str, parameter_set_name, parameter_set_class)

fill a template block with a set of paramters, input : keys - a list of the parameters config_fixed - the fixed parameters config_instance - the parameters of the instance config_default - the default parameters tree_str - current position in the object tree (parent.child. …) parameter_set_name - name of the parameter_set parameter_set_class - parameter_set class

getHtml()

generate html

get_config_dependencies(obj)

retrieve all config dependencies of obj

makeHtmlTree(tree_list)

from the tree_list make a html tree

str_par_types = {<class 'int'>: 'int', <class 'str'>: 'str', 'list-int': 'list-int', 'list-float': 'list-float', <class 'float'>: 'float'}
str_to_object = {'float': <class 'float'>, 'int': <class 'int'>, 'list-float': <function HtmlProcessParameters.<lambda> at 0x7f932bb76620>, 'list-int': <function HtmlProcessParameters.<lambda> at 0x7f932bb76598>, 'str': <class 'str'>}
walk_obj_tree(obj_instance, obj_default, tree_str='', parent=None)

make a html tree view from an object, recursive

astro.services.process2.HtmlProcessParameters.detectCircularRecursion(parent, child, target)

Detect a circular (endless) recursion loop

class astro.services.process2.HtmlProcessParameters.parBase

Bases: object

base class for the custom classes defining process parameters

get_onthefly_dependencies(config=None)
classmethod is_cal()
classmethod is_science()
class astro.services.process2.HtmlProcessParameters.parGAS

Bases: astro.services.process2.HtmlProcessParameters.parBase

class defining the process parameters for the Gastrom (task)

astromconf = <astro.main.Config.AstromConfig object>
classmethod get_fixed_config()
classmethod get_persistent_properties()
class astro.services.process2.HtmlProcessParameters.parGastromSourceList

Bases: astro.services.process2.HtmlProcessParameters.parBase

class defining the process parameters for the GastromSourceList (task)

classmethod get_default_config()
classmethod get_fixed_config()
classmethod get_persistent_properties()
sexconf = <astro.main.Config.SextractorConfig object>
astro.services.process2.HtmlProcessParameters.update_config(config_obj, config)

update the config object with the key / values in the config dict

astro.services.process2.HtmlProcessPreferences module

File HtmlProcessPreferences.py Author WJ Vriend (wjvriend[AT]astro.rug.nl) Date Jul 2007

class astro.services.process2.HtmlProcessPreferences.HtmlProcessPreferences(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

classmethod arg_handler(parent)

extract the set preferences by the user from args

classmethod check_config(pref_dict)

Check the given config dict, fill missing elements with default values. Method does not check the content of preferences ! (content checking is done in the arg_handler)

default_dpu = 'dpu.hpc.rug.astro-wise.org'
default_privileges = 1
derived_supported = {('AstrometricParameters', 'reduced'): ['ReducedScienceFrame'], ('CoaddedRegriddedFrame', 'regridded_frames'): ['RegriddedFrame'], ('ColdPixelMap', 'flat'): ['DomeFlatFrame', 'TwilightFlatFrame'], ('FringeFrame', 'flat'): ['MasterFlatFrame'], ('PhotSrcCatalog', 'frame'): ['ReducedScienceFrame'], ('PhotometricParameters', 'extinct'): ['AtmosphericExtinctionCoefficient'], ('SourceList', 'detection_frame'): ['ReducedScienceFrame', 'RegriddedFrame', 'CoaddedRegriddedFrame', 'RadioFrame'], ('SourceList', 'frame'): ['ReducedScienceFrame', 'RegriddedFrame', 'CoaddedRegriddedFrame', 'RadioFrame']}
doDerived()

Get the html for setting the derived classes Derived info is stored as name=value pairs :

class_name.attribute=class_name
doPreferences()

generate the html for the preferences

classmethod getDefaultsDict()

return a dictionary with the default preferences

getHtml()

generate html

classmethod getParallelDpuHtml(preferences, extra='', test=False, id_name='pref_dpu_parallel')

get the html selection box for the parallel dpu

classmethod getSingleDpuHtml(preferences, extra='', test=False, id_name='pref_dpu_single')

get the html selection box for the single dpu

classmethod get_default_preferences(defaults_only=False)

Return the default preferences structure preference name : ( description, order, default), When defaults_only==True return only filled in defaults, otherwise return full preference dict

classmethod get_ordered_preferences()

return the preferences ordered

classmethod try_privileges(parent, privileges)

try the given privileges

astro.services.process2.HtmlProcessPreferences.getDPUsHtml(selected, min_cpu=-1, max_cpu=-1, max_len=30, test=False)

make html select options for all clusters (dpu’s)

astro.services.process2.HtmlProcessSL4AL module

File HtmlProcessSL4AL.py Author WJ Vriend (wjvriend@astro.rug.nl)

Generates html page for selecting Sourcelists for AssociateList

class astro.services.process2.HtmlProcessSL4AL.HtmlProcessSL4AL(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

classmethod arg_handler(parent)

store the returned arguments in the master_dict

getHtml()

generate HTML using template

store_vars = ['SLIDS', 'SLID', 'ra', 'dec', 'PM', 'object_name', 'input_method']
class astro.services.process2.HtmlProcessSL4AL.HtmlProcessSL4AL_Result(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

class for showing the resulting SourceList’s

getHtml()

generate html

class astro.services.process2.HtmlProcessSL4AL.HtmlProcess_TargetView_AL(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

specialized HtmlProcess_TargetView class for generating html for AssociateLists

MAX_SOURCELISTS = 1000
classmethod arg_handler(parent)

store the chosen sourcelist as targets in the master_dict

fillCataloginTemplate(tmpl, catalogs)

set the list of Catalogs in the template

fillSourceListInTemplate(tmpl, sl, nr)

fill single SourceList in template

fillSourceListsInTemplate(tmpl, SLs, exclude_all_sky=True)

set the list of SourceLists in the template

generateSL()

generate list of SourceLists based on input

generateSLorder()

generate html to let the user set the order of the SourceLists

getHtml()

generate html

classmethod getSLids(args)

get the (selected) SourceList ids from the arguments

row_classes = ['TableRowEven', 'TableRowOdd']
setError(msg)

set error message

store_vars = ['associate_type']
astro.services.process2.HtmlProcessSL4AL.convert_arg(parent, arg, arg_value, convert_type)
astro.services.process2.HtmlProcessSL4AL.makeSLCoordQuery(ra_min, ra_max, dec_min, dec_max)

return query for SourceList based on 4 coordinates

astro.services.process2.HtmlProcessSL4AL.orderSLQuery(qry)

Order the SourceList query by …

astro.services.process2.HtmlProcessSelectSids module

File HtmlProcessSelectSids.py Author WJ Vriend (wjvriend[AT]astro.rug.nl) Class to render html pages for selecting SIDS from a SourceList

class astro.services.process2.HtmlProcessSelectSids.HtmlProcessSelectSids_Input(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

classmethod arg_handler(parent)

handle the the SLID input

getHtml()

generate html

intro_page()

show the intro page, with the SLID input

store_vars = ['SLID']
class astro.services.process2.HtmlProcessSelectSids.HtmlProcessSelectSids_Query(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

classmethod arg_handler(parent)

handle the queries set by the user

getHtml()

show the query page, the user can add query conditions based on the sourcelist attributes

store_vars = ['queries', 'sids', 'SLID']
classmethod use_queries(slid, queries, sl_dict)

use the queries given by the user and return sourcelist result

class astro.services.process2.HtmlProcessSelectSids.HtmlProcessSelectSids_Result(parent, argsa)

Bases: common.services.general.HtmlBase.HtmlBase

classmethod arg_handler(parent)

handle the queries set by the user

getHtml()

generate html

result_page(tmpl)

show the resulting sources based on the queries

result_page_sids(tmpl)

show the resulting sources using the given SID list

astro.services.process2.HtmlProcessSubmit module

HtmlProcessSubmit.py generates html pagte to submit target processor jobs

WJ Vriend

class astro.services.process2.HtmlProcessSubmit.HtmlProcessSubmit(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

class plots a window with the task(s) to be submitted

MAX_TARGETS_BEFORE_MSG = 200
classmethod arg_handler(parent)

the user has send the job to the dpu

default_process_time = 1799
getHtml()

generate html for job submit page

get_chip_string()

get the string to represent the chips

classmethod initSubmit(parent, depth_cal, depth_sci, group_chip, nr_of_cpus, GAS, use_synchronization, privileges, codefile, job_label, job_dict)

initialize OnTheFlyJobSubmit

initTemplate(tmpl)

set the variables in the template

max_jobs()

determine the maximum number of jobs to run parallel

single_dpu_jobs = ('CoaddedRegriddedFrame', 'Gastrometric')
store_vars = ('targets', 'job_submitted')
astro.services.process2.HtmlProcessSubmit.getFilterDisplay(targets)

from list of filters make string to display

astro.services.process2.HtmlProcessSubmit.get_datetime_display(targets, timezone=None)

from list of datetimes make string to display if timezone is set getDayValue of datetime

astro.services.process2.HtmlProcess_TargetView module

File HtmlProcess_TargetView.py Author WJ Vriend (wjvriend@astro.rug.nl)

class astro.services.process2.HtmlProcess_TargetView.HtmlProcess_TargetView(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

default HtmlProcess_TargetView class for generating html

HTML_ERROR = '<html><body><h1>Error</h1><h2>%s</h2></body></html>'
Hdr_Chip = 'Chip'
These are two main modes :
1 Objects with a chip attribute are plotted for all chips on one line
Example; Raw, Reduced, Regrid, Bias, ColdPixel …
2 Objects without a chip are plotted one object per line
Example; CoaddedRegriddedFrame, input for MasterFlatFrame
Mode 1 has two sub modes :
1A Targets with a single input; Reduced, Regrid, … 1B Targets with multiple inputs; Bias, Fringe, …

When there are multiple inputs a table for each targets is shown where the user can select which raws will be used for each target.

Special cases :
Hot/ColdPixel; the input for Cold is RawDome and for Hot RawBias. MasterFlatFrame; the user has no choice, only the input for the MasterFlat is shown

For Coadds existing objects are shown for reprocessing.

Hdr_Filename = 'Filename'
Hdr_Filter = 'Filter'
Hdr_Observation = 'Observation'
MASTERFLAT_HEADER = '\n<h3>MasterFlatFrame</h3>\n<p>\nSelect whether the MasterFlatFrame should be made with a TwilightFlat, DomeFlat and/or NightSkyFlatFrame.\nFurther below are the details of the frames that will be used as input.\n</p>\n<table class="ProcessTable">\n <tr><td><b>Date</b><td>%s - %s\n <tr><td><b>Filter</b><td>%s\n <tr><td><input type="checkbox" checked="checked" name="use_dome" id="mycb-target-dome"><td><b>DomeFlatFrame (%d)</b>\n <tr><td><input type="checkbox" checked="checked" name="use_nightsky" id="mycb-target-nightsky"><td><b>NightSkyFlatFrame (%d)</b>\n <tr><td><input type="checkbox" checked="checked" name="use_twilight" id="mycb-target-twilight"><td><b>TwilightFlatFrame (%d)</b>\n</table>\n<p><input type="button" value="Query" class="ProcessButton" onclick="do_query_set(1,0);">\n&nbsp; &nbsp; &nbsp; or &nbsp; &nbsp; &nbsp;\n<input type="button" value="Process" class="ProcessButton" onclick="do_process_set(1,0);"></p>\n'
MAX_RESULTS = 5000
append_row(row1, row2)

append the current row ?

classmethod arg_handler(parent)

handle the given arguments, extract the targets for the submit page or the TreeView (query) page

compare_rows(row1, row2)

compare the date_obs, filter, grid, and filename of two rows

doCalibration(tmpl)

plot the to be made and existing calibration (like) frames these frames can have multiple input raw frames

doCoadd(tmpl)

plot new Coadds and existing ones

doMasterFlat(tmpl, fix_dep_by_oid=False)

plot the existing master frames and the input frames for the master flat

doScience(tmpl)

plot new and existing science (like) frames frames with a date_obs and dependency to one raw frame

doSourceList(tmpl)

plot SourceLists to be made, and existing ones

getChipHtmlString(chip, fltr, is_valid, quality_flags, filename, date_obs)

depending on the flag color background and link to image server

getDependencies(object_id)

get the dependencies dict for the target

getFilterName(fltr)

get the filter name from the filter ID / short name

getHeadingNew()

construct the heading text

getHtml()

generate html

getListsFromRs(rs)

find the newest (creation_date) version of all chips and concatenate set is ordered by chip, creation_date

getProcessedDateChipsTableRows(date_obs, fltr, table, chip_list=[])

for the specified table make html table rows for each creation_date (processed) and chips given the date_obs

getProcessedHtml(date_obs, fltr, chips)

make the table with the processed info for these raw

getProcessedRS()

get the recordset with processed objects

getSqlExisting_Coadd()

get the sql for selecting the existing coadds

getSqlExisting_MasterFlat()

Query for existing (processed) MasterFlatFrame dependencies self.ProcessedClass should have been set

getSqlExisting_RadioFrame()

get the sql for selecting the existing RadioFrame’s

getSqlNew()

the default query for selecting raw objects (used for new objects)

getSqlWhere(table, cls, raw_table='')

given the class cls construct a where statement with asrguments

classmethod get_grouped(processed_type, raw_type)

return whether the obj_str is grouped per

get_masterflat_header()

make heading for the MasterFlatFame

index_grouped_per_day = 2
index_grouped_per_filter = 3
index_processed = 1
index_supported = 0
iter_rows(block, rows, table_headers)

iterate the rows and fill the template

make_processed_dict(processed_rs)

from the processed recordset make a structured (processed) dictionay

oracle_date_format = 'YYYY-MM-DD HH24:MI:SS'
plotDetails_ChipsPerRow(block, raw_rs, table_headers, heading='')

Plot the rows in the recordset (rs) All chips of one observation are plotted in one row of the html table

plotDetails_OnePerRow(rs, table_headers, html_block, heading='', p_existing_buttons_extra='')

Plot the rows in the recordset (rs) These objects will be plotted one per row

plotRow(block_details, row_nr, header_nr, rows)

construct the variables for a row in the html table

plotRowInTemplate(block_details, row, row_nr, header_nr, opener_id, processed, chip)

plot a row in the html table

plot_date_format = '%d %b %Y %H:%M:%S'
python_date_format = '%Y-%m-%d %H:%M:%S'
same_block(row1, row2)

compare rows to see if we need a new block, check day (!), filter and grid

setTypeClass(processed_type)

from the process_type set self type and class for processed and raw only used for MasterFlat target

skip_row(row1, row2)

skip the current row ?

startNewBlock(block, table_headers, header_nr)

print the table header info

store_vars = []
targets = {'AstrometricParameters': (1, 1, 0, 0), 'BiasFrame': (1, 1, 1, 0), 'CoaddedRegriddedFrame': (1, 0, 0, 1), 'ColdPixelMap': (1, 1, 1, 1), 'DarkCurrent': (1, 0, 1, 0), 'DomeFlatFrame': (1, 1, 1, 1), 'FringeFrame': (1, 1, 1, 1), 'GainLinearity': (1, 0, 1, 0), 'HotPixelMap': (1, 1, 1, 0), 'MasterFlatFrame': (1, 0, 0, 0), 'PhotSrcCatalog': (1, 1, 0, 0), 'PhotometricParameters': (1, 0, 0, 0), 'ReadNoise': (1, 1, 1, 0), 'ReducedScienceFrame': (1, 1, 0, 0), 'RegriddedFrame': (1, 1, 0, 0), 'SourceList': (1, 0, 0, 0), 'TwilightFlatFrame': (1, 1, 1, 1)}
exception astro.services.process2.HtmlProcess_TargetView.ProcessTargetViewError

Bases: astro.services.process2.Process.ProcessError

class astro.services.process2.HtmlProcess_TargetView.SqlGenerator(cls, select, sql_from, from_alias, where, order_by, union=None)

Bases: object

generate_from(cls)

generate the FROM clause

generate_orderby()

generate the ORDER BY clause

generate_select()

generate the SELECT clause

generate_where()

generate the WHERE clause

getSql()

generate SQL

joins = {'astrom': (None, 'astrom.object_id = t.object_id'), 'astrom_params': (<class 'astro.main.AstrometricParameters.AstrometricParameters'>, 'astrom_params.object_id = t.astrom_params'), 'chip': (<class 'astro.main.Chip.Chip'>, 'chip.object_id = t.chip'), 'filter': (<class 'astro.main.Filter.Filter'>, 'filter.object_id = t.filter'), 'grid_target': (<class 'astro.main.RegriddedFrame.GridTarget'>, 'grid_target.object_id = t.grid_target'), 'instrument': (<class 'astro.main.Instrument.Instrument'>, 'instrument.object_id = t.instrument'), 'observing_block': (<class 'astro.main.ObservingBlock.ObservingBlock'>, 'observing_block.object_id = t.observing_block'), 'template': (<class 'astro.main.ObservingBlock.Template'>, 'template.object_id = t.template')}
astro.services.process2.HtmlProcess_TargetView.get_select_clause(cls, check_attr, select_attr, default)

return the select clause for attr (with name), depending on class

astro.services.process2.HtmlProcess_TargetView.schema_table(cls)

construct table name with schema

astro.services.process2.HtmlProcess_TreeView module

File HtmlProcess_TreeView.py Author WJ Vriend (wjvriend@astro.rug.nl)

class astro.services.process2.HtmlProcess_TreeView.HtmlProcessTree(instrument_obj, *args, **kwargs)

Bases: common.services.process.HtmlProcessTree.HtmlProcessTree

class generates the object tree, showing the state of all dependencies specific astro version

data_server = 'http://ds.astro.rug.astro-wise.org:8000'
formatObjectForErrors(obj, all_objects)

Return datetime of object, this can be timestamps, date_obs or creation_date

getObjectOverviewTableElements(uptodate_dict, elem)

return the name,value pairs for the object overview table

getOnTheFlyValue(elem, txt_tree)

(overridden) determine onthefly status if status is ONTHEFLY_FIXED_DEP, then elem must also be present in fixed_dependencies

classmethod makeObjectString(obj)

(overridden) make an object specific string with datetime/chip/filter with cls.object_id_sep as seperator

skipObject(parent_class, obj)

(overridden) if obj is None or not selected chip, skip object

class astro.services.process2.HtmlProcess_TreeView.HtmlProcess_TreeView(parent, args)

Bases: common.services.general.HtmlBase.HtmlBase

MAX_OBJECT = 100
MAX_TARGET_BEFORE_NOFLY = 50
classmethod arg_handler(parent)

handle the UI parameters of the HtmlProcess_TreeView screen

date_format_python = '%Y-%m-%d'
fill_statistics(tmpl, stats)

finish the template, filling in statistics

getHtml()

generate html

makeTreeHeader()

set the header for this tree

time_format_python = '%H:%M:%S'

astro.services.process2.OnTheFlyJob module

OnTheFlyJob’s for the Target Processor

main job is OnTheFlyJob, that contains SubJob’s

Specific targets have specific derived SubJob’s

class astro.services.process2.OnTheFlyJob.OnTheFlyJob

Bases: common.net.Jobs.Job

class for holding multiple (sub) jobs, which will be run on the same node

addSubJob(sub_job)

append a sub job

countSubJob()

return the number of sub jobs

execute()

execute the job

exception astro.services.process2.OnTheFlyJob.OnTheFlyJobError

Bases: Exception

class astro.services.process2.OnTheFlyJob.OnTheFlySubJob(obj_type=None, obj_id=None, attributes=None, config=None)

Bases: astro.services.process2.OnTheFlyJob.SubJob

class defines the parameters for a job

execute()

print stuff and call get_onthefly

class astro.services.process2.OnTheFlyJob.OnTheFlySubJob_AL(obj_type=None, obj_id=None, config=None)

Bases: astro.services.process2.OnTheFlyJob.SubJob

OnTheFly sub Job for AssociateLists

execute()

make an AssociateList of type 1 or 2 with more then 2 input lists

class astro.services.process2.OnTheFlyJob.OnTheFlySubJob_GAS(target_ids, instrument, pars=None, config=None)

Bases: astro.services.process2.OnTheFlyJob.OnTheFlySubJob_GAS_base

OnTheFly sub Job for GAS

execute()

call the GAstromSourceListTask

class astro.services.process2.OnTheFlyJob.OnTheFlySubJob_GAS_SL(target_ids, instrument, raiseException=True, pars=None, red_filenames=None, config=None)

Bases: astro.services.process2.OnTheFlyJob.OnTheFlySubJob_GAS_base

OnTheFly sub Job for GAS SourceList’s

countSubJob()

return the number of sub jobs

execute()

call the GAstromSourceListTask

class astro.services.process2.OnTheFlyJob.OnTheFlySubJob_GAS_base(config=None)

Bases: astro.services.process2.OnTheFlyJob.SubJob

base class for the GAS jobs

getReducedFilenames()

from the regrid ids get the reduced filenames the ids can be a tuple (date, chip, filter) or object_ids

class astro.services.process2.OnTheFlyJob.OnTheFlySubJob_GalFit(obj_type=None, obj_id=None, config=None)

Bases: astro.services.process2.OnTheFlyJob.SubJob

OnTheFly sub Job for GalFitModels

execute()

call the GalPhotTask

class astro.services.process2.OnTheFlyJob.OnTheFlySubJob_GalPhot(obj_type=None, obj_id=None, config=None)

Bases: astro.services.process2.OnTheFlyJob.SubJob

OnTheFly sub Job for GalPhotModels

execute()

call the GalPhotTask

class astro.services.process2.OnTheFlyJob.SubJob(config=None)

Bases: object

base class for a sub job

config = None
classmethod convert_process_parameters_to_task(parameters, source_par, dest_par)

convert the process parameters to the task format process parameters check if the parameter key starts with source_par and replace with dest_par source_par; name of the source parameters, example GAstromSourceList dest_par ; name of the destination parameters, example SourceList

countSubJob()

return the number of sub jobs, default 1

execute()

method must be implemented by derived class

obj_type = None

astro.services.process2.OnTheFlyJobSubmit module

OnTheFlyJobSubmit.py

class for submitting OnTheFly jobs

WJ Vriend

class astro.services.process2.OnTheFlyJobSubmit.JobGenerator(parent)

Bases: object

class for generating jobs a dedicated helper class for OnTheFlySubmit

eatParameters(input_parameters, target_attr, target_cls, base=0)

Get the process parameters for RegriddedFrames (and lower) targets from those specified for a Coadd like target. Dependending on the target eat the first N (base + to_eat) target names in the parameter dependency string : parent.child1.child2… If the first target name (after eating) matches target_attr, replace this with target_cls.

estimate_time(**args)

forward estimate time to request to parent

getAstrometricParametersJobs(regrids_per_chip, target, base=1)

make AstrometricParameter jobs from regridded frames in target

getCoadd_SL_Job(target)

make a coadd and or sourcelist job from regrids_deps / coadd dep

getGasJob(regrids_per_chip, parameters)

make a GAS job from reduced filenames

getGasSlJobs(regrids_per_chip, parameters)

initialize OnTheFlySubJob_GAS_SL jobs

getRegridJobs(regrids_per_chip, target)

make regrid jobs from regridded frames in target

class astro.services.process2.OnTheFlyJobSubmit.OnTheFlyJobSubmit

Bases: object

DEFAULT_TIME = 250
DPU_AWETARGET = 'astro'
JOBS_GROUP = 0
JOBS_SPECIFY = 2
JOBS_SPREAD = 1
OptimizeJobs(jobs, mode, job_name)

For INDEPENDENT jobs make job_dicts with list of N jobs, where N is the number of cores per node

TEST = True
addJob(target)

add a job, get the parameters form the target dict

default_counts = {'AssociateList': 5, 'BiasFrame': 20, 'CoaddedRegriddedFrame': 40, 'DarkCurrent': 20, 'DomeFlatFrame': 20, 'FringeFrame': 20, 'GAstrometric': 40, 'NightSkyFlatFrame': 20, 'PhotRedCatalog': 10, 'SourceList': 10, 'TwilightFlatFrame': 20}
dpu_minimal_time = 600
dpu_time_multiplier = 1
estimate_time(level=None, input_count=None, object_type=None)

estimate the time needed for processing

getCode(codefile)

Get the code object, this is a zipped file with the (whole) python code tree

classmethod getCoresPerJob(dpu_name, task, default_cores_per_job=1)

return the number of cores per job for the given dpu name and task

classmethod getJobLayout(obj_type, GAS=False)

return the job layout given the target object type, and optional GAS

getJobs()

for all the given targets create a dictionary of subjobs ordened by chip

getJobsCoadd()

as getJobs, but for Coadd, SourceList and AssociateList synchronization points are taken into account only for Coadd (at the moment)

getJobsGAS(ids_per_chip, target, base=1)

return jobs for AstrometricParameters, GAS plus SourceLists and GAS

get_level()

return the (process) level, with a ceiling

groupJobs(jobs_per_chip, mode, job_name, time=250, info_str=None)

make a list of job dicts using a jobs_per_chip dict as input

initJob(obj_type=None, config=None, codefile=None, group_chip=0, nr_of_cpus=1, privileges=0, test_mode=False, use_synchronization=True, job_label='', job_dict=None)

initialize a job

makeJobDict(jobs=None, info=None, mode=None, time=None, job_name=None, task=None)

make a dpu job dict, dpu dictionary layout : { ‘DPU_JOBS’ : N * [jobs],

‘DPU_MODE’ : mode, ‘DPU_TIME’ : time in seconds, ‘DPU_INFO’ : N * [job name], ‘DPU_NAME’ : dpu name }
makeJobInfo(obj_type, chip)

make a job info string

process_time_per_cal_level = 150
sendJob()

make and send the jobs

set_subjob_class()

determine and set the subjob_class attribute

submitJobs(dpu, jobs)

submit jobs to the dpu

astro.services.process2.OnTheFlyJobSubmit.copyParameters(input_parameters, target)

copy the parameter (sets) that start with target

astro.services.process2.OnTheFlyJobSubmit.getDPU(dpu_name)

return dpu with name dpu_name

astro.services.process2.OnTheFlyJobSubmit.getEnv()

get a environment structure overriding the defaults

astro.services.process2.OnTheFlyJobSubmit.getMaxJobs(jobs_per_chip)

get the maximum number of (sub) jobs per chip

astro.services.process2.OnTheFlyJobSubmit.get_grid_target(regrid_ids, instrument)

Determine the most common grid target from the regrid ids The regrid ids can be (date, chip, filter) or object_id

astro.services.process2.OnTheFlyJobSubmit.makeCode(filename)

Read in the code-zip file (filename), and return as string var

astro.services.process2.OnTheFlyJobSubmit.makeGalList(cls, SLID, name, commit=True)

make a GalPhotList or GalFitList

astro.services.process2.OnTheFlyJobSubmit.reorderPerChip(elem_list, obj_type)

reorder all elements per chip, input is list of tuples where [1] is chip

astro.services.process2.OnTheFlyJobSubmit.translateObjectIds(objects_per_chip, obj_type)

translate an obj per chip dict from object_ids to datetime/chip/filter

astro.services.process2.Process module

File HtmlProcess_TargetView.py Author WJ Vriend (wjvriend[AT]astro.rug.nl)

general methods for the Target Processor

exception astro.services.process2.Process.ProcessError

Bases: Exception

astro.services.process2.Process.fillFilterDict(instrument)

fill the filter dictionary for the given instrument

astro.services.process2.Process.formatDate(input_date)

format a date to a date string

astro.services.process2.Process.formatDateTime(input_date)

format a date to a datetime string

astro.services.process2.Process.getEmptyDependencyDict(instrument, dictionary=True)

for the given instrument return a dictionary with for each chip an empty dictionary or list

astro.services.process2.Process.getFilterName(fltr, instrument=None)

loop through the tuple (id, name) looking for fltr returning name

astro.services.process2.Process.getTarget(object_type, datetime, chip, fltr)

get the (latest) target of object_type using datetime/chip/filter log failure

astro.services.process2.Process.get_codefile_name(username)

from the username, make the (zipped code) filename

astro.services.process2.Process.get_pixel_scale()

get the deault pixel scale for grid_target

astro.services.process2.Process.get_url(server='process.astro-wise.org', project=None, privileges=None)

return the url of the target processor

astro.services.process2.Process.get_url_dbview(table='', object_id='', project='', privileges=1, object=None, coadd_reprocess_regrids=True, server='process.astro-wise.org')

return url for dbview web service to reprocess an object

astro.services.process2.Process.get_url_reprocess(table='', object_id='', project='', privileges=1, object=None, coadd_reprocess_regrids=True, server='process.astro-wise.org')

return url for dbview web service to reprocess an object

astro.services.process2.Process.logDependencyDict(dependency_dict, p=False)

log the self.dependency_dict dictionary

astro.services.process2.Process.logJob(job, p=False)

log individual job, print most attributes

astro.services.process2.Process.logJobs(jobs, p=False)

log (and print) the jobs to be submitted

astro.services.process2.Process.logParameterDict(parameter_dict, p=False)

log the self.parameter dictionary

astro.services.process2.Process.log_print(text, to_print)

log and print text

astro.services.process2.Process.master_dict_astrometry_GAS(master_dict)

translate the master_dict astrometry variable to GAS switch

astro.services.process2.Process.translateOnTheFly(onthefly, caldepth, imgdepth)

translate onthefly value (from process web page) to calibration depth (caldepth)

astro.services.process2.TargetView_Input module

Handle the input from the TargetView

WJ Vriend

target is a dictionary specifying the target to make, with keys required :

class : target class
required to specify object :
chip : the chip datetime : the observation date filter : the filter
OR specify object with :
object_id : object_id of object to process
optional :
dependencies : the dependencies dictionary, default {} parameters : the process parameters, default {}
class astro.services.process2.TargetView_Input.TargetView_Input(args, master_dict)

Bases: object

class converts the options in the target view window to targets

dependency_strings_extra = {'ColdPixelMap': 'raw_domeflat_frames', 'HotPixelMap': 'raw_bias_frames'}
getArg(arg, default='')

get an argument from the list

getExistingTargets(target_class)

get the selected targets from the dictionary targets_existing

getGroupedTargets(target_class)

get the selected targets and the selected input for the targets for grouped targets the user has choses the input targets from a set

getMasterFlatTargets(target_class)

get the target dictionary for the master flat specifying which flat dependency should NOT be used

getNonGroupedTargets(target_class)

get targets from target_dict, these are not grouped, so there is no choice for multiple raw frames

getSourceListTargets()

get the targets for the sourcelist

classmethod getTargetDict(target_class=None, chip='', date=None, filter='', object_id='')

return the dictionary describing the target

get_target_sets()

get the selected targets

get_targets()

return the target(s) from the arguments in the target view window

astro.services.process2.TargetView_Input.getDepedenciesExistingTarget(target_class, object_id, reprocess_coadd_regrids=True)

get the dependencies for an existing target, for a cadd these are the regrids, for a sourcelist these are a coadd

astro.services.process2.TargetView_Input.splitInput(dependency_identifiers)

input is [datetime, filter, chip|obj_id 1, chip|obj_id 2, …] return datetime, filter, [chips], [obj_ids]

astro.services.process2.TreeView_Input module

Handle the input from a treeview

There are the following options :
choose which tree to process view all chips of a certain tree
In the past there was the possibility to:
set dependencies to None fix dependencies
class astro.services.process2.TreeView_Input.TreeView_Input(args, master_dict)

Bases: object

class converts the options in the treeview window to targets

eatDepStr(dep_str, number)

eat the first number entries from dep_str

getIdAndDepStr(dep_str_ID)

dep_str_ID = Parent.Dependecy[ID].reduced. … ID = object_id or datetime|chip|filter

get_targets()

return the target(s) from the arguments in the treeview window

handle_args()

handle the user input

makeIdFromList(elems)

elems = object_id or date (%Y%m%d), time(%H%M%S), chip, filter

setDependencyCoadd(full_dep_str, dep_id)

place fixed dependencies (below regrid) in seperate dictionary dict and fixate regrids by dep_id

setDependencyNone(dep_str, dep_chip)

set the dependency to None

translateDependencyId(ID, dep_class_str, chip)

if ID set, then find similar dependency for given chip

Module contents