Common architectural design for persistency of Galfit, Galphot, PhotZ, VODIA, MDIA, etc.

Introduction

This section describes the architectural design of a common structure to store output of programs like Galfit, Galphot and PhotZ. Other packages of interest are VODIA and MDIA, which require a slightly different design. In the case of Galfit and Galphot, source parameters are derived based on a SourceList (and source identifier) input. In the case of PhotZ, redshifts are derived based on more than one SourceList. VODIA and MDIA work on RegriddedFrames.

Consequently, these programs can be divided into two categories:

  • Programs that use SourceList(s) as input: Galfit, Galphot, PhotZ
  • Programs that use RegriddedFrame(s) as input: VODIA, MDIA

Design for programs that use SourceList(s) as input

Galfit and Galphot calculate a number of parameters for a particular source. This is conceptually identical to what Sextractor does. However, Sextractor (through the SourceList class) is run on an image, which presumably contains many sources. Galfit and Galphot are run on a per-source basis. The process of running e.g. Galfit on a per-source basis is parallellized by obtaining a cutout for each source and sending different cutouts to different CPUs/machines to be processed. It is possible to run e.g. Galfit a (potentially large) number of times to find optimal initial parameters.

Common base class design

Galfit, Galphot, and PhotZ have the common property that they calculate one or more parameters that describe a source. To link these values to a particular source, the following structure is the miminum necessity:

class BaseSourceLink(DBObject):
    SLID = persistent('SourceList identifier', int, 0)
    SID = persistent('Source identifier', int, 0)
    is_valid = persistent('Flag to disqualify bad data', int, 1)
    creation_date = persistent('', datetime.datetime, datetime.datetime(1990,1,1))

The BaseSourceLink class provides what is common in the designs for Galfit, Galphot and PhotZ.

Galfit

For Galfit two main classes exist: GalFitComponent and GalFitModel. GalFitModels are galaxy models that can consist of more than one component (a bulge and a disk, for example). Additional components are added as necessary for foreground stars. GalFitComponent is a parent class for a number of classes, one for each fitted function type: GalFitSersic, GalFitDevauc, GalFitNuker, etc:

class GalFitModel(BaseSourceLink, GalFitProduct):
    SLID =
    SID =
    GFID =
    is_valid =
    creation_date =
    components = persistent('', GalFitComponent, [])
    process_params = persistent('', GalFitParameters, None)
    region_center_x
    region_center_y
    region_width
    region_height
    chi2 =
    dof =
    rchi2 =

class GalFitComponent(BaseSourceLink, GalFitProduct):
    SLID =
    SID =
    GFID =
    name = # Sersic, Devauc etc.
    primary = # main object or star to mask (model)
    Z = # subtract model

class GalFitSersic(GalFitComponent):
    x       = persistent('Fitted value of x position', float, 0.0)
    ix      = persistent('Initial value of x position', float, 0.0)
    dx      = persistent('Error in fitted x position', float, 0.0)
    free_x  = persistent('Fix or free (fit) the parameter', int, 1)
    y       = persistent('Fitted value of y position', float, 0.0)
    iy      = persistent('Initial value of y position', float, 0.0)
    dy      = persistent('Error in fitted y position', float, 0.0)
    free_y  = persistent('Fix or free (fit) the parameter', int, 1)
    etc.

Examples of Galfit queries

  • Query among GalFitModels to find those with sersic fit and which are

primary objects.

q = (GalFitComponent.name == 'sersic') & (GalFitComponent.primary == 1) &\
    (GalFitComponent.SLID==57424)
gfids = [c.GFID for c in q]
galfitmodels=[GalFitModel.GFID==c.GFID for c in q]
print len(gfids),len(galfitmodels)
  • Find all GalFitModels for sources in a given SLID for which ra1

\(<\) Ra \(<\) ra2 and dec1 \(<\) Dec \(<\) dec2

sl=(SourceList.SLID==57424)[0]
r = sl.sources.area_search(Area=[(9.46,-29.30),(9.46,-29.20),(9.45,-29.20),\
                                 (9.45,-29.30)])
print len(r[sl.SLID])

l = []
for sid in r[sl.SLID]:
    q = GalFitModel.SID == sid
    if len(q):
        for m in q:
            l.append(m)
print len(l)
print l
  • From a bunch of frames with a bunch of sourcelists (e.g., mosaic of

cluster of galaxies): find all GalFitModels made for these sourcelists, and modelled with a sersic profile which has an exponential N \(<\) 2.0 plus a sky component.

qsl=SourceList.name.like('GAS-Sci-GSIKKEMA*')
print len(qsl)

slids = [sl.SLID for sl in qsl]
lsersic=[]
for slid in slids:
    q = GalFitModel.SLID == slid
    for m in q:
        qsersic = (GalFitSersic.GFID == m.GFID) & (GalFitSersic.N < 2.0)
        qsky=(GalFitSky.GFID == m.GFID)
        if (len(qsersic)==1 & len(qsky)==1):
                p = 0
                for c in m.components:
                    p += c.primary
                if p == 2:
                    lsersic.append(qsersic[0])
print lsersic
print len(lsersic)
  • Starting from an AssociateList find the Sersic N for the sourcelists

of a R and B image for which an associatelist exists.

al =(AssociateList.ALID ==9367)[0]

slids=[sl.SLID for sl in al.sourcelists]
attrlist = ['SLID','SID','RA', 'DEC']
r = al.associates.get_data(attrlist,mask=3,mode='ALL')
aids = [k for k in r.keys()]
aids.sort()
print 'AID', attrlist
slid_B=slids[0]
slid_R=slids[1]
r_e={slid_B:[],slid_R:[]}
for aid in aids[:10]:
    for row in r[aid]:
        q = (GalFitSersic.SLID == row[0]) & (GalFitSersic.SID == row[1])
        for c in q:
            print c.N
            r_e[row[0]].append([row[1],c.N])
print r_e

Galphot

For Galphot, a structure similar to Galfit may be possible. In particular the idea is that the ellipse shape parameters can be stored in a structure similar to GalFitComponents:

class GalPhotEllipse(BaseSourceLink):
    SLID =
    SID =
    GPID =
    r
    dr
    i
    e_i
    s
    e_s
    x
    e_x
    y
    e_y
    eps
    e_eps
    pos
    e_pos
    c1
    e_c1
    etc.

class GalPhotModel(BaseSourceLink)
    SLID =
    SID =
    GPID =
    is_valid =
    creation_date =
    ellipses = persistent('', GalPhotEllipse, [])
    etc.

PhotZ

PhotZ calculates photometric redshifts. This is done by creating what is essentially a low resolution Spectral Energy Distribution for each source, one point for each observation in a different filter.

Associating the same (physical) source as observed through different filters, takes the form of an AssociateList in Astro-Wise. The AssociateList associates sources (identified by the “SID” identifier) in different SourceLists (identified by the “SLID” identifier).

The current design of PhotZ creates an output SourceList with redshift information which is associated to the input AssociateList to form a new AssociateList. This is the final output of PhotZ. The input of PhotZ is a catalog with combined aperture photometry values for different filters (?).

It is possible to store those redshifts (and other output of PhotZ) one source at a time in a structure that is similar to that of Galfit/Galphot, however. Storing the information this way involves copying data (for each filter the same redshifts will be stored in different BaseSourceLink objects), but it will make the data easier to query.

class PhotZData(BaseSourceLink): # i.e. DATPZ1 data
    SLID =
    SID =
    is_valid =
    creation_date =
    RA =
    DEC =
    Xpos =
    Ypos =
    A =
    B =
    POSANG =
    obj = # Object ID
    best_z = # redshift
    err_z = # Error
    mod = # model ID
    rchi2 = # reduced chi2
    z2 = # second best fitting z
    lg_Pz2z1 = # ration of the probabilities of z2/z
    <z> = # weighted mean of the z distribution
    fU, fB, fV, fR, fI, fJ, fK, fF1, fF2, fF3, fF4, fF5, fF6, fF7,
    M_B, M_R, M_I, M_K # fluxes of object and derived absolute magnitudes
    DMOD = # distance modulus
    f_dat/f_mod = # ratio between observed/model flux
    best_model = # best fitting model

# Note: DatStar sourcelist
class PhotZsomething(BaseSourceLink):
    is_valid =
    creation_date =
    RA
    DEC
    Xpos
    Ypos
    A
    B
    POSANG
    obj = # object ID
    best_star = # Model ID of best fitting star
    rchi2 = # reduced chi2
    fU, fB, fV, fR, fI, fJ, fK, fF1, fF2, fF3, fF4, fF5, fF6, fF7 # object fluxes
    f_dat/f_mod = # ratio between observed/model flux

Design for programs that use RegriddedFrames as input

Both the VODIA and the MDIA packages do Difference Image Analysis to detect variable objects and to calculate their light curves. For both packages similar Python classes can be made. A desired functionality of lightcurves is that of adding points to the curve when extra observations are made. A class with the attributes RegriddedFrame (input), x, y, and an identifier defining the light curve to which a (additional) point belongs should allow for this.

VODIA

The input of VODIA:

  • list of RegriddedFrames
  • set of configurations (see awe/astro/experimental/VODIO/Dia.py)
  • list of best images to produce the ref. image
  • list of stars used as PSF-reference
  • list of stars for which a lightcurve is produced

The output of VODIA can be easily recognized from the following classes, which serve as the main data containers for VODIA output:

class VODIALightCurvePoint:
    VODIA_ID = 1 # different LightCurvePoints that form a curve have the same ID
    regridded = RegridddedFrame # filename?
    (filename)
    is_valid =
    creation_date =
    x
    y
    JD
    HJD
    BJD
    RA
    DEC
    PSF_phot
    err_PSF_phot
    ap_phot
    err_ap_phot
    PSF_phot_diff
    err_PSF_phot_diff
    ap_phot_diff
    err_ap_phot_diff
    bg
    chi2_PSF
    corr_PSF
    chi2_kern
    fwhm
    nbad
    flag

class VODIALightCurve:
    VODIA_ID =
    is_valid =
    creation_date =
    regridded_frames =    # transient as this can change by adding a point
    reference = BaseFrame
    mask = PixelMap
    stars_lightcurve = [] # does not change when adding a point to the curve
    stars_psf = []        # does not change when adding a point to the curve
    mstackconf =
    getpsfconf =
    agaconf =
    getvarconf =
    photconf =

Adding a point to a lightcurve is then as simple as committing another VODIALightCurvePoint object, which has the same VODIA_ID as the existing ones, but a different RegriddedFrame dependency. That is provided that the reference image does not change. If the reference image does change, a new LightCurve object must be derived.

Also the following classes would have to be stored to store the configuration parameters etc.

class VODIAmstackconf(Config):
    ...
class VODIAgetpsfconf(Config):
    ...
etc.

MDIA

The MDIA class structure would be virtually identical to that of VODIA:

  • list of RegriddedFrames
  • set of configurations
  • list of best images
  • list of stars to be masked
  • list of stars used as PSF-reference
  • list of stars for photometrical alignment
  • list of stars for which a lightcurve is produced
class MDIALightCurvePoint:
    MDIA_ID =
    regridded = RegriddedFrame
    (filename)
    is_valid =
    creation_date =
    x
    y
    JD
    HJD
    BJD
    RA
    DEC
    PSF_phot
    err_PSF_phot
    chi2_PSF
    nbad
    flag

class MDIALightCurve:
    MDIA_ID =
    is_valid =
    creation_date =
    regridded_frames = # transient!
    process_params =
    best_images =
    stars_mask = []
    stars_psf = []
    stars_align = []
    stars_lightcurve = []
    reference = BaseFrame
    reference_error = BaseFrame