astro.toolbox.ingest.libs package

Submodules

astro.toolbox.ingest.libs.DBOverviewCASU module

Script to get the contents of the CASU database for the VIKING project.

Output is an ASCII table

class astro.toolbox.ingest.libs.DBOverviewCASU.DBOverviewCASU

Bases: object

get(query='179.A-2004', sql='is_tile=true')
read_table()

Filename Chip Object_Name Coords Obs_date Band Airmass Exptime SeeingEllipt ZP ZPerr Maglim WCS rms Version OB_Status ESO_Grade Last_Updated OB_Name

astro.toolbox.ingest.libs.DBOverviewCASU.main()

astro.toolbox.ingest.libs.DataRequestCASU module

class astro.toolbox.ingest.libs.DataRequestCASU.DataRequestCASU

Bases: object

request(email, username, password, filename, reqtype)
astro.toolbox.ingest.libs.DataRequestCASU.main()

astro.toolbox.ingest.libs.IngestClassifier module

astro.toolbox.ingest.libs.OBGenerator module

exception astro.toolbox.ingest.libs.OBGenerator.OBGeneratorError

Bases: Exception

astro.toolbox.ingest.libs.OBGenerator.group_rawfitsdata(rfd_list, image_type, force_single_template=0)

Groups object frames (science and photom) first by day, then by object and then by filter, groups filter independent calibration frames (readnoise, bias, dark, and gain) by day only, and groups filter dependent calibration frames (dome and twilight) first by day, then by filter.

This type of grouping allows the flexibility of multiple templates (e.g., sorted by DATE_OBS or by filter) in one observing block (e.g., with a single OBJECT or for a single day)

astro.toolbox.ingest.libs.OBGenerator.set_observingblock(rfd_groups, image_type)

This routine is used to create an ObservingBlock object from the lists (representing separate templates) of RawFitsData objects contained within the “rfd_groups” input list. This is then assigned to the ‘observing_block’ attribute of the RawFitsData objects. It is assumed that all the input RawFitsData objects are from one and the same observing night, and (if appropriate) have all been observed in one and the same filter.

NOTE: The input list of RawFitsData objects is filtered to remove any
that contain an ObservingBlock already committed to the database. Those that contain a valid ObservingBlock are untouched, and those that remain are used to construct a new ObservingBlock and have this new one set to their observing_block attribute.
astro.toolbox.ingest.libs.OBGenerator.set_template(rfd_group, image_type)

This routine is used to create a Template object from the RawFitsData objects contained within the “rfd_group” input list. This is then assigned to the ‘template’ attribute of the RawFitsData objects. It is assumed that all the input RawFitsData objects are from one and the same observing night, and (if appropriate) have all been observed in one and the same filter.

NOTE: The input list of RawFitsData objects is filtered to remove any
that contain a Template already committed to the database. Those that contain a valid Template are untouched, and those that remain are used to construct a new Template and have this new one set to their ‘template’ attribute.

astro.toolbox.ingest.libs.RawDataConverter module

Convert RawData to RawFitsData.

class astro.toolbox.ingest.libs.RawDataConverter.RawDataConverter(filenames=[], memmap=1)

Bases: object

Class used to convert RawData (e.g., single-extension FITS images, or compressed multi-extension FITS images) to RawFitsData (i.e., multi-extension FITS, or MEF images).

filenames: list of RawData filenames to convert
memmap: use memory mapping (where applicable) when converting
NOTE: Memory mapping is essential on limited RAM machines when converting
large mosaics and is enabled by default. If it causes problems, try disabling it.
check_preconditions()

Are there images to be converted?

create_rawfitsdata()

Convert SEF images into MEF images.

group_by_date_obs()

This method is used to group a list of rawdata frames by date_obs.

make()
sort_frames()
exception astro.toolbox.ingest.libs.RawDataConverter.RawDataConverterError

Bases: Exception

astro.toolbox.ingest.libs.RawDataConverter.get_basename(filenames)
astro.toolbox.ingest.libs.RawDataConverter.get_distinct_date_obs(input_frames)
astro.toolbox.ingest.libs.RawDataConverter.get_filenames(input_frames)
astro.toolbox.ingest.libs.RawDataConverter.translate_date_keys(frame)

astro.toolbox.ingest.libs.RawFitsDataIngestor module

exception astro.toolbox.ingest.libs.RawFitsDataIngestor.IngestorError

Bases: Exception

class astro.toolbox.ingest.libs.RawFitsDataIngestor.RawFitsDataIngestor(filenames, image_type, commit=False, force=False, force_invalid_template=False, force_single_template=False)

Bases: object

This ingestor is used for ingesting data from most instruments.

ingest()
class astro.toolbox.ingest.libs.RawFitsDataIngestor.RawFitsDataIngestorESO(filenames, image_type, commit=False, force=False, force_invalid_template=False, force_single_template=False)

Bases: astro.toolbox.ingest.libs.RawFitsDataIngestor.RawFitsDataIngestor

This ingestor is used for ingesting data from ESO instruments.

ingest()
class astro.toolbox.ingest.libs.RawFitsDataIngestor.RawFitsDataIngestorVIRCAM(filenames, image_type, commit=False, force=False, force_invalid_template=False, force_single_template=False)

Bases: astro.toolbox.ingest.libs.RawFitsDataIngestor.RawFitsDataIngestorESO

This ingestor is exclusively used for ingesting VIRCAM data.

ingest()

Ingest the “filenames”.

Note:
  1. In the case of stacks for each filename a RawFitsData object for the weight is also created.
  2. In the case of single pawprints weights are present for longer time periods, like a month or 15 days, so an extra few weights RawFitsData objects will be created.
astro.toolbox.ingest.libs.RawFitsDataIngestor.create_rawfitsdata_ingestor(instrument_name)
astro.toolbox.ingest.libs.RawFitsDataIngestor.distinct(alist)

Returns the distinct values from an input list.

astro.toolbox.ingest.libs.RawFitsDataIngestor.store_and_commit(obj)

This function calls the store() and commit() methods of “obj”. It also excepts the “Unique constrait violation error” to write a more informative message.

astro.toolbox.ingest.libs.RawFrameIngestor module

exception astro.toolbox.ingest.libs.RawFrameIngestor.IngestorError

Bases: Exception

class astro.toolbox.ingest.libs.RawFrameIngestor.RawFrameIngestor

Bases: object

Abstract base class to ingest RawFrames from RawFitsData

check_data()

Compare the input RawFitsData atributes to known values.

compress_store_commit_rawframes()
create_raw_frames()

Make the RawFrame objects (includes splitting into single frames).

derive_astrometric_correction()

If the data needs it, run an astrometric correction.

ingest(rawfitsdata, image_type, commit=0)
set_default_config(rawframe)

Set instrument specific default config properties

class astro.toolbox.ingest.libs.RawFrameIngestor.RawFrameIngestorGeneric

Bases: astro.toolbox.ingest.libs.RawFrameIngestor.RawFrameIngestor

class astro.toolbox.ingest.libs.RawFrameIngestor.RawFrameIngestorOCAM

Bases: astro.toolbox.ingest.libs.RawFrameIngestor.RawFrameIngestor

astro.toolbox.ingest.libs.RawFrameIngestor.create_rawframe_ingestor(instrument_name)
astro.toolbox.ingest.libs.RawFrameIngestor.store_and_commit(obj)

This function calls the store() and commit() methods of “obj”. It also excepts the “Unique constrait violation error” to write a more informative message.

astro.toolbox.ingest.libs.ReducedFrameIngestor module

class astro.toolbox.ingest.libs.ReducedFrameIngestor.BaseReducedFrameIngestor

Bases: object

Assemble some useful methods used by the ReducedScienceFrame ingestor and other derived ingestors.

check_data()

Compare the input RawFitsData atributes to known values.

create_calibrations()

Make the calibration objects for each reduced frame. Currently, this derives the Astrometric and PhotometricParameters objects as dictated by the HeaderTranslator methods.

derive_astrometric_correction()

If the data needs it, run an astrometric correction.

store_commit_redframes()
exception astro.toolbox.ingest.libs.ReducedFrameIngestor.IngestorError

Bases: Exception

class astro.toolbox.ingest.libs.ReducedFrameIngestor.ReducedFrameIngestor

Bases: astro.toolbox.ingest.libs.ReducedFrameIngestor.BaseReducedFrameIngestor

Class to ingest ReducedFrames from RawFitsData

create_red_frames()

Make the RedFrame objects (includes creating WeightFrame).

ingest(rawfitsdata, image_type, weight='', commit=False)
make_redframe(frame)
set_persistent_attributes(frame)

Set attributes specific to this RedFrame and its instrument.

class astro.toolbox.ingest.libs.ReducedFrameIngestor.ReducedFrameIngestorISAAC

Bases: astro.toolbox.ingest.libs.ReducedFrameIngestor.BaseReducedFrameIngestor

Ingests ISAAC data into the database as ReducedScienceFrames. ISAAC data is for only one CCD. Its RawFitsData therefore has no extensions, so a single ReducedScienceFrame is created.

copy_persistent_attributes_from_rawfitsdata()

Set attributes specific to this RedFrame and its instrument.

create_redframe()

Make the ReducedScienceFrame objects (includes creating WeightFrame).

ingest(rawfitsdata, image_type, commit=0)
store_commit_redframe()
class astro.toolbox.ingest.libs.ReducedFrameIngestor.ReducedFrameIngestorVIRCAM

Bases: astro.toolbox.ingest.libs.ReducedFrameIngestor.BaseReducedFrameIngestor

Ingestor for reduced frames of VIRCAM

The standard ReducedFrameIngestor assumes the weights are incorporated in the reduced science file. For VIRCAM the weights (“confidence maps”) are found in separate files.

copy_persistent_attributes_from_rawfitsdata(frame)
create_comments_reduced(frame)
create_comments_weight(frame, weight)
create_red_frames()

Make the RedFrame objects (includes creating WeightFrame).

exists_in_database(redframe)

Check if a ReducedScienceFrame object exists in the database for the same DATE-OBS, chip name and CASUVERS.

Returns True or False

ingest(rawfitsdata, image_type, commit=0)
make_redframe_part1(frame)
make_redframe_part2(frame)
set_astrom(frame)
set_chip(frame)
set_naxis(frame)
set_photom(frame)
set_weight(frame)
astro.toolbox.ingest.libs.ReducedFrameIngestor.create_reducedframe_ingestor(instrument_name)
astro.toolbox.ingest.libs.ReducedFrameIngestor.store_and_commit(obj)

This function calls the store() and commit() methods of “obj”. It also excepts the “Unique constrait violation error” to write a more informative message.

Module contents

This directory contains the various classes that make up the ingest subsystem. These are not for direct use.