common.util package

Submodules

common.util.Compressor module

Provide a common interface to common compression utilities: one using Python modules, the other using shell programs.

class common.util.Compressor.BZipFile(filename)

Bases: common.util.Compressor.GZipFile

compressor = 'bzip2'

ext = '.bz2'

class common.util.Compressor.GZipFile(filename)

Bases: object

Simple (de)compression using a commandline tools.

compress(level=6, verbose=False, force=False, keep=False)

compressor = 'gzip'

decompress(verbose=False, force=False, keep=False)

ext = '.gz'

class common.util.Compressor.ZipFile(in_filename, out_filename=None, compression=None, level=6, verbose=True, clobber=False, keep=False, buffer_size=128)

Bases: object

Advanced (de)compression using Python modules.

check_preconditions()

Make sure the input file exists.

create_file_objects()

Create the file objects to be used in the compression process. This is the meat of the class. Here, the decisions are made regarding how the compression will be done as this needs to be determined prior to creating the file objects.

The philosophy of the gzip.GzipFile and bz2.BZ2File classes is that they open a GZip or BZip2 file for reading or writing as if it were a normal file. Any (de)compression is done transparently. This allows for a simpler programming interface as the classes need no extra methods called except read() and write().

make()

Convert input file to output file.

type_map = ['gz', 'bz2']

write_and_close_files()

Do the actual file I/O.

exception common.util.Compressor.ZipFileError

Bases: Exception

common.util.Compressor.is_bz2(file)

common.util.Compressor.is_gz(file)

common.util.OptionsParser module

Simple options parser, meant to parse arguments to tasks

class common.util.OptionsParser.OptionsParser(name='')

Bases: object

Simple options parser, meant to parse arguments to tasks

add_option(short_name, long_name, default, doc, mandatory=False)

Add an option to the OptionsParser.

Example: parser.add_option(‘i’, ‘instrument’, ‘’, ‘The instrument’)

expand_options(*args, **kwargs)

Convert any short options into their long options

get_options()

parse(*args, **kwargs)

Parse the options specified in keyword=value format

common.util.OptionsParser.same_type(lhs, rhs)

Help comparison of ints and bools

common.util.Pipeline module

The following common base classes are defined:

TaskError PipelineError Task CompositeTask Pipeline SequentialOptions

class common.util.Pipeline.CompositeTask

Bases: common.util.Pipeline.Task

This is a composite task. It iterates over its “task_list” and feeds the output of each step as input to the next task.

add(task)

check_preconditions()

check_process_result(process_result)

clean_up_before_pickle()

This method is intended to prevent both pickle errors and triggering problems with the Oracle object cache leading to erroneous query results in the AWE prompt.

execute()

get_process_time()

get_result()

Returns the combined get_result() from all Tasks in self.task_list

get_task_info()

return the task as info string, used for DPU_INFO

iterate_over_tasklist()

remove()

class common.util.Pipeline.Pipeline

Bases: object

This abstract class represents a generic pipeline.

CPJ_DICT = {}

MEM_DICT = {}

atomic_task_map = {}

classmethod atomic_tasks()

awetarget = 'common'

contains_single_node_task(task_sequence)

This boolean method checks whether a given sequence of task identifiers contains a single_node task. The method is private.

classmethod create_paralleloptions(pipeline, options)

classmethod create_pipeline(sequence_identifier)

classmethod get_available_sequence_identifiers()

get_group_size(task_key, ntasks)

Based on the following factors:

1. Number of nodes demanded by user
2. Number of processes per node
3. Number of cores per job

Returns the best group size for the task “task_key”.

classmethod get_instance()

classmethod get_joblist(sequence_identifier, ppnode=1, nnodes=1, dpu_time=0, dpu_mem=0, dpu_cpj=0, dpu_aweversion='', dpu_name='', **options)

Returns a dictionary which is the input to the DPU and describes the jobs to be executed.

ppnode means processes per node; dual core/CPU machines handle 2 processes simultaneously

classmethod get_sequence_options(sequence_identifier)

Return a list of options recognized by the sequence identifier

classmethod get_sequence_params(sequence_identifier)

Return process_params as dict for sequence

classmethod get_tasks_of_sequence(sequence_identifier)

classmethod has_sequence_identifier(sequence_identifier)

classmethod sequence_identifiers()

classmethod show_sequence_options(sequence_identifier)

Print a readable list of options for the first task in the sequence

single_node_tasks = []

task_sequences = {}

to_joblist(sequence_identifier, **options)

This method is used to “dump” the ImagePipeline to a job list. The method is public.

exception common.util.Pipeline.PipelineError(message)

Bases: common.log.Error.Error

Pipelines’ own exceptions

class common.util.Pipeline.SequentialOptions(options, pipeline, task_sequence)

Bases: object

In a composite entry, i.e. Reduce>Coadd, for each step the same options will be used. However they can be filtered to exclude all options that are not understood by (any of) the next processing steps. I.e. the overscan correction option that can be used for Reduce can better be removed for the subsequent steps.

get_sequential_options()

Return a list of option dictionaries, suitable for each step of the task_sequence. Options not recognised by a step are discarded for that step

class common.util.Pipeline.Task(*args, **kwargs)

Bases: object

This is the abstract base class from which every processing task derives.

clean_up_before_pickle()

override in derived Task classes, see CompositeTask for description

execute()

This method is used to actually run the processing task. The method is abstract.

classmethod filter_options(options)

Returns a dictionary containing those options in “options” that are recognized by Task ‘cls”.

get_arguments()

Return (args, kwargs), potentially updated, given upon instantiation.

classmethod get_options()

get_process_result()

This method is used to retrieve the result from executing the task. This result should be a list of instances of one of the many sub-types of DBObject. The method is abstract

get_process_time()

Calculates the needed process time in seconds. This method must be overridden in the derived class

get_result()

Returns a list of objects which are the results of the processing done by this task. Note that objects must have been committed for this method to work.

get_task_info()

return the task as info string, used for DPU_INFO

parser = Recognized options for the OptionsParser are:

set_dependencies(dependencies)

classmethod show_options()

target = None

update_options(*args, **kwargs)

Update the keyword arguments in the kwargs property overwriting existing keyword arguments, then update the options property with this updated kwargs dictionary.

args : options list

non-keyword arguments

kwargs : keyword options dict

keyword arguments

use_process_result(process_result_entry)

Use a process result, defaults to using self.target (class) and process_result entry as object_id

exception common.util.Pipeline.TaskError(message)

Bases: common.log.Error.Error

Tasks’ own exceptions

common.util.Processor module

The Processor and LocalProcessor base classes

Derived class should define the Pipeline attribute

class common.util.Processor.BaseProcessor

Bases: object

Pipeline = None

get_sequence_identifiers()

Returns a list of all known sequence identifiers.

A sequence identifier is the first argument given to the dpu.run() method. It defines which task(s) is (are) executed.

get_sequence_options(sequence)

Returns all recognized options for the first task in the sequence of tasks defined by the sequence identifier.

get_sequence_params(sequence)

Returns all process parameters and their values for the sequence.

show_sequence_identifiers()

Print the list of all known sequence identifiers

(Sequence identifiers are strings such as: “Reduce” or “Reduce>Regrid”)

show_sequence_options(sequence)

Print all recognized options for the first task in the sequence of tasks defined by the sequence identifier.

class common.util.Processor.LocalProcessor

Bases: common.util.Processor.BaseProcessor

This class is used as a frontend to “run” processes (tasks) on the local machine in the same way jobs are run using the dpu.

get_job_result()

Returns the objects created by the last task/job

run(sequence_identifier, **options)

Send a job or a sequence of jobs to be processed.

sequence_identifier – identifier of the task or sequence of tasks (e.g.

‘Bias’ or ‘Reduce>Astrom’)

options – arguments given to the task(s) that will be run

exception common.util.Processor.LocalProcessorError(message)

Bases: common.log.Error.Error

class common.util.Processor.Processor(dpu_name, request_logs=False)

Bases: common.util.Processor.BaseProcessor

This class is used as a frontend to “run” processes (tasks) in a distributed fashion, for example on the HPC cluster in Groningen. This class uses the HTTP frame work (“Distributed Processing Unit”) coded for this purpose (see common/net/dpu*).

The Processor makes a connection to the specified DPU, schedules tasks for processing and then starts these processes. Output messages from the DPU may be printed on standard out, and written to a log file in the local directory.

A Processor can be started by giving as argument a DPU identifier, i.e.:

awe> dpu = Processor(‘dpu.hpc.rug.astro-wise.org’)

Note that this is done automatically at awe startup.

Processes can then be started by calling the “run” method and supplying a task identifier argument and options for that task:

awe> dpu.run(‘ReadNoise’, i=’OMEGACAM’, tpl=‘2014-12-26T08:56:37’,

dpu_time=1799, send_code=False, commit=False)

Logs for finished jobs can be collected by calling the “get_logs” method:

awe> dpu.get_logs()

cancel_job(jobid=None)

Delete a key or cancel a (queued) job.

Multiple calls may be necessary in some circumstances

dpu_aweversion = ''

get_dpu_identifiers()

Returns the list of all DPUs which can be used to process data.

A DPU identifier may be used to instantiate a Processor instance:

awe> dpu = Processor(identifier)

get_job_result(jobid)

Returns the objects created by the task/job with the specified jobid

get_jobids()

Returns a list of active jobids

get_log(jobid, quiet=False)

get_logs(jobids=None, quiet=False)

Get logs for finished jobs or the job identifier(s) specified. Logs are printed on stdout and written to file.

jobids: list of job IDs

quiet: quiet default behavior of not writing logs to screen

get_onelog(jobid, jobnum=0, quiet=False)

Retrieve the log of a single (sub-)job, including stderr and stdout

get_status(jobids=None)

Get status for all jobs or the job identifier(s) specified.

jobids: list of job IDs

The line returned is similar to:

Jobid 91421 has status Q/R/F/E/A/S/U 60/4/0/0/0/0/0

FINISHED N/E/A/S/U 64/0/0/0/0

where:

Q = QUEUED : # of jobs in the queue R = RUNNING : # of jobs currently being processed F = FINISHED : # of jobs finished processing N = NORMAL : # of jobs finished normally A = ABORTED : # of jobs aborted by the cluster E = ERROR : # of jobs ending with an error S = SKIPPED : # of jobs skipped U = USERABORT : # of jobs aborted by the user

get_syslogs(jobid, quiet=False)

Get the server side openpbs/glite system logs (stderr and stdout)

info()

is_finished(jobid)

parse_jobids(jobids)

run(sequence_identifier, dpu_time=0, dpu_mem=0, dpu_cpj=0, dpu_aweversion='', send_code=None, nnodes=0, **options)

Send a job or a sequence of jobs to be processed.

sequence_identifier – identifier of the task or sequence of tasks,

(see dpu.show_sequence_identifiers())

dpu_time – override any dpu_time (expected processing time)

derived by the tasks themselves

dpu_mem – specifying the memory required may influence

which nodes are used

dpu_aweversion – version (AWBASE or current) of binaries to

use on the compute cluster.

options – arguments given to the task(s) that will be run

(see dpu.show_sequence_options(seq_id))

send_code – send your Python code (located in directory

AWEPIPE to use on the DPU)

nnodes – demand nnodes for the job at hand

select_dpu(dpu_name)

Select a different dpu to use

send_code = True

set_dpu_client(dpu_name)

show_dpu_identifiers()

Print the list of DPUs which can be used to process data

wait_for_jobs(jobids=None, poll=5, maxwait=-1, maxwait_start=None)

Wait for either the specified jobs, or all your known jobs to finish. This method is useful when using the dpu / Processor class in a script.

jobids: list of job identifiers

poll: polling frequency in seconds

maxwait: max time to wait in seconds, if exceeded raise ProcessorError

exception common.util.Processor.ProcessorError(message)

Bases: common.log.Error.Error

common.util.Processor.check_excessive_number_of_jobs(joblist, nmax=1000)

Count the total number of jobs inside the joblist and refuse to process them if the total is more than nmax (default 1000). Goal is not to overload the queueing system.

common.util.Processor.estimate_total_process_time(joblist)

Group jobs by type: ‘IND’, ‘SEQ’, ‘SYN’. Then take max DPU_TIME for all IND jobs (they run independently of anything) and add all SEQ and SYN jobs.

common.util.ShellCommand module

Functions to run commands as a subprocess / shell commands

common.util.ShellCommand.is_known_command(command)

A hackish way to find if a given command exists

common.util.ShellCommand.pipecommand(command)

Returns tuple of retcode (int), exitstatus (int), output (list of output lines)

common.util.ShellCommand.pipecommand_err(command)

Returns tuple of retcode (int), exitstatus (int), output (list of output lines), errors (list of errors)

Also close_fds=True

common.util.ShellCommand.run(command, suppress=1, verbose=1)

A utility function to run shell commands

Parameter suppress is used to quiet down the process.

Returns a value != 0 if the command failed.

common.util.URL module

URL.py functions for URL parsing

common.util.URL.combineURI(protocol, domain, port=None, basepath=None, filename=None)

accept protocol, domain, port, path,filename return URL protocol://domain[:port][/path/filename]

common.util.URL.parseURI(URI)

accept string as URI and parses it to protocol, domain, port, path,filename accepted form protocol://domain[:port][/path/filename]

common.util.Xml2Object module

Xml to Python object(s) parser

Xml file (or string) is parsed and ouput is a Python object Element, which can have Element children, etc … The xml attributes and character data is stored in Element.

Example :

parser = Xml2Object( ) root_element = parser.Parse(‘example.xml’)

original code:

see http://code.activestate.com/recipes/149368/

updated:

Element.getAllElements

class common.util.Xml2Object.Element(name, attributes)

Bases: object

A parsed XML element

addChild(element)

Add a reference to a child element

getAllElements(element_array, name='')

Get an array of all elements recursively

getAttribute(key)

Get an attribute value

getData()

Get the cdata

getElement(name)

Get the element with name, if a list present return the first

getElements(name='')

Get a list of child elements

hasElement(name)

return True if there is a child with given name, otherwise False

hasElementLike(name)

return True if there is a child which contains name, otherwise False

setAttribute(key, value)

Set an attribute value

setData(data)

Set the data

toString(level=0, indent=' ')

print element and children

class common.util.Xml2Object.Xml2Object(encoding=None, strip_data=True)

Bases: object

XML to Object converter

CharacterData(data)

Expat character data event handler

EndElement(name)

Expat end element event handler

Parse(filename, xml=None)

Create an Expat parser, input can be a filename or the xml

StartElement(name, attributes)

Expat start element event handler

exception common.util.Xml2Object.Xml2ObjectError

Bases: Exception

common.util.awePickle module

a wrapper around the pickle module for use in awe

class common.util.awePickle.awePickle(filename, protocol=2)

Bases: object

a wrapping of the pickle module and functions to (un)pickle an object (from) to a file

pickle(obj)

pickle obj to the given filename

unpickle()

unpickle from the given filename

common.util.imports_2to3 module

Module with imports which differ between the Python 2.x and 3.x environment

All imports should import the module names the same for 2.x and 3.x

common.util.imports_2to3.get_charset_from_response(response)

common.util.install module

class common.util.install.CVSModule(dirname, checkoutcommand, buildcommand, installcommand)

Bases: object

build(show_only=0)

get_filename()

has_been_downloaded()

has_been_unpacked()

make()

retrieve(show_only=0)

unpack(show_only=0)

class common.util.install.GITModule(dirname, checkoutcommand, buildcommand, installcommand)

Bases: common.util.install.CVSModule

class common.util.install.Package(t, urlname, buildcommand, installcommand)

Bases: object

build(show_only)

get_filename()

has_been_downloaded()

has_been_unpacked()

make()

retrieve(show_only=0)

unpack(show_only=0)

common.util.install.getarchitecture()

common.util.install.geturlcommand()

common.util.install.insidevirtualenv()

Return True if we are running inside a virtualenv or venv environment, False otherwise.

common.util.install.makeinstallscript(packagenames=[], cvsmodules=[], gitmodules=[], variables={})

called from the various awetomatic scripts as

makeinstallscript(packagenames=packagenames, variables=globals())

or

makeinstallscript(packagenames=packagenames, cvsmodules=cvsmodules, variables=globals())

where packagenames looks like this:

packagenames = [

# swig [ (‘swig’, ‘1.3.40’) , downloadprefix + ‘%(name)s/%(name)s-%(version)s.tar.gz’ , ‘’‘env AWEVERSION=%(aweversion)s ./configure –with-python %(aweexecutable)s –prefix=%(aweprefix)s make’‘’ , ‘make install’ ], …

]

common.util.md5sum module

common.util.md5sum.md5sum(file, length=16384)

common.util.path module

path.py - An object representing a path to a file or directory.

Example:

from path import path d = path(‘/home/guido/bin’) for f in d.files(‘*.py’):

f.chmod(0755)

This module requires Python 2.2 or later.

URL: http://www.jorendorff.com/articles/python/path Author: Jason Orendorff <jason@jorendorff.com> (and others - see the url!) Date: 23 Feb 2003

class common.util.path.path

Bases: str

Represents a filesystem path.

For documentation on individual methods, consult their counterparts in os.path.

abspath()

access(mode)

Return true if current user has access to this path.

mode - One of the constants os.F_OK, os.R_OK, os.W_OK, os.X_OK

atime

Last access time of the file.

basename(p)

Returns the final component of a pathname

bytes()

Open this file, read all bytes, return them as a string.

chmod(mode)

chown(uid, gid)

chroot()

copy(src, dst, *, follow_symlinks=True)

Copy data and mode bits (“cp src dst”). Return the file’s destination.

The destination may be a directory.

If follow_symlinks is false, symlinks won’t be followed. This resembles GNU’s “cp -P src dst”.

If source and destination are the same file, a SameFileError will be raised.

copy2(src, dst, *, follow_symlinks=True)

Copy data and all stat info (“cp -p src dst”). Return the file’s destination.”

The destination may be a directory.

If follow_symlinks is false, symlinks won’t be followed. This resembles GNU’s “cp -P src dst”.

copyfile(src, dst, *, follow_symlinks=True)

Copy data from src to dst.

If follow_symlinks is not set and src is a symbolic link, a new symlink will be created instead of copying the file it points to.

copymode(src, dst, *, follow_symlinks=True)

Copy mode bits from src to dst.

If follow_symlinks is not set, symlinks aren’t followed if and only if both src and dst are symlinks. If lchmod isn’t available (e.g. Linux) this method does nothing.

copystat(src, dst, *, follow_symlinks=True)

Copy all stat info (mode bits, atime, mtime, flags) from src to dst.

If the optional flag follow_symlinks is not set, symlinks aren’t followed if and only if both src and dst are symlinks.

copytree(src, dst, symlinks=False, ignore=None, copy_function=<function copy2>, ignore_dangling_symlinks=False)

Recursively copy a directory tree.

The destination directory must not already exist. If exception(s) occur, an Error is raised with a list of reasons.

If the optional symlinks flag is true, symbolic links in the source tree result in symbolic links in the destination tree; if it is false, the contents of the files pointed to by symbolic links are copied. If the file pointed by the symlink doesn’t exist, an exception will be added in the list of errors raised in an Error exception at the end of the copy process.

You can set the optional ignore_dangling_symlinks flag to true if you want to silence this exception. Notice that this has no effect on platforms that don’t support os.symlink.

The optional ignore argument is a callable. If given, it is called with the src parameter, which is the directory being visited by copytree(), and names which is the list of src contents, as returned by os.listdir():

callable(src, names) -> ignored_names

Since copytree() is called recursively, the callable will be called once for each directory that is copied. It returns a list of names relative to the src directory that should not be copied.

The optional copy_function argument is a callable that will be used to copy each file. It will be called with the source path and the destination path as arguments. By default, copy2() is used, but any function that supports the same signature (like copy()) can be used.

ctime

Creation time of the file.

dirname()

dirs() → List of this directory's subdirectories.

The elements of the list are path objects. This does not walk recursively into subdirectories (but see path.walkdirs).

With the optional ‘pattern’ argument, this only lists directories whose names match the given pattern. For example, d.dirs(‘build-*’).

drive

The drive specifier, for example ‘C:’. This is always empty on systems that don’t use drive specifiers.

exists(path)

Test whether a path exists. Returns False for broken symbolic links

expand()

Clean up a filename by calling expandvars(), expanduser(), and normpath() on it.

This is commonly everything needed to clean up a filename read from a configuration file, for example.

expanduser()

expandvars()

ext

The file extension, for example ‘.py’.

files() → List of the files in this directory.

The elements of the list are path objects. This does not walk into subdirectories (see path.walkfiles).

With the optional ‘pattern’ argument, this only lists files whose names match the given pattern. For example, d.files(‘*.pyc’).

fnmatch(pattern)

Return True if self.name matches the given pattern.

pattern - A filename pattern with wildcards,

for example ‘*.py’.

getatime(filename)

Return the last access time of a file, reported by os.stat().

getctime(filename)

Return the metadata change time of a file, reported by os.stat().

static getcwd()

Return the current working directory as a path object.

getmtime(filename)

Return the last modification time of a file, reported by os.stat().

getsize(filename)

Return the size of a file, reported by os.stat().

glob(pattern)

Return a list of path objects that match the pattern.

pattern - a path relative to this directory, with wildcards.

For example, path(‘/users’).glob(‘/bin/’) returns a list of all the files users have in their bin directories.

isabs(s)

Test whether a path is absolute

isdir(s)

Return true if the pathname refers to an existing directory.

isfile(path)

Test whether a path is a regular file

islink(path)

Test whether a path is a symbolic link

ismount(path)

Test whether a path is a mount point

joinpath(*args)

Join two or more path components, adding a separator character (os.sep) if needed. Returns a new path object.

lines(encoding=None, errors='strict', retain=True)

Open this file, read all lines, return them in a list.

Optional arguments:

encoding - The Unicode encoding (or character set) of

the file. The default is None, meaning the content of the file is read as 8-bit characters and returned as a list of (non-Unicode) str objects.

errors - How to handle Unicode errors; see help(str.decode)

for the options. Default is ‘strict’

retain - If true, retain newline characters; but all newline

character combinations (‘

‘, ‘ ‘, ‘ ‘) are

translated to ‘

‘. If false, newline characters are

stripped off. Default is True.

This uses ‘U’ mode in Python 2.3 and later.

link(newpath)

Create a hard link at ‘newpath’, pointing to this file.

listdir() → List of items in this directory.

Use D.files() or D.dirs() instead if you want a listing of just files or just subdirectories.

The elements of the list are path objects.

With the optional ‘pattern’ argument, this only lists items whose names match the given pattern.

lstat()

Like path.stat(), but do not follow symbolic links.

makedirs(mode=511)

mkdir(mode=511)

move(src, dst, copy_function=<function copy2>)

Recursively move a file or directory to another location. This is similar to the Unix “mv” command. Return the file or directory’s destination.

If the destination is a directory or a symlink to a directory, the source is moved inside the directory. The destination path must not already exist.

If the destination already exists but is not a directory, it may be overwritten depending on os.rename() semantics.

If the destination is on our current filesystem, then rename() is used. Otherwise, src is copied to the destination and then removed. Symlinks are recreated under the new name if os.rename() fails because of cross filesystem renames.

The optional copy_function argument is a callable that will be used to copy the source or it will be delegated to copytree. By default, copy2() is used, but any function that supports the same signature (like copy()) can be used.

A lot more could be done here… A look at a mv.c shows a lot of the issues this implementation glosses over.

mtime

Last-modified time of the file.

name

Returns the final component of a pathname

normcase()

normpath()

parent

pathconf(name)

readlink()

Return the path to which this symbolic link points.

The result may be an absolute or a relative path.

readlinkabs()

Return the path to which this symbolic link points.

The result is always an absolute path.

realpath()

relpath()

Return this path as a relative path, based from the current working directory.

relpathto(dest)

Return a relative path from self to dest.

If there is no relative path from self to dest, for example if they reside on different drives in Windows, then this returns dest.abspath().

remove()

removedirs()

rename(new)

renames(new)

rmdir()

rmtree(path, ignore_errors=False, onerror=None)

Recursively delete a directory tree.

If ignore_errors is set, errors are ignored; otherwise, if onerror is set, it is called to handle the error with arguments (func, path, exc_info) where func is platform and implementation dependent; path is the argument to that function that caused it to fail; and exc_info is a tuple returned by sys.exc_info(). If ignore_errors is false and onerror is None, an exception is raised.

samefile(f1, f2)

Test whether two pathnames reference the same actual file

size

Size of the file, in bytes.

splitall()

Return a list of the path components in this path.

The first item in the list will be a path. Its value will be either os.curdir, os.pardir, empty, or the root directory of this path (for example, ‘/’ or ‘C:’). The other items in the list will be strings.

path.path.joinpath(*result) will yield the original path.

splitdrive()

splitext()

splitpath() → Return (p.parent, p.name).

stat()

Perform a stat() system call on this path.

statvfs()

Perform a statvfs() system call on this path.

symlink(newlink)

Create a symbolic link at ‘newlink’, pointing here.

text(encoding=None, errors='strict')

Open this file, read it in, return the content as a string.

This uses ‘U’ mode in Python 2.3 and later, so ‘

‘ and ‘ ‘

are automatically translated to ‘

‘.

Optional arguments:

encoding - The Unicode encoding (or character set) of

the file. If present, the content of the file is decoded and returned as a unicode object; otherwise it is returned as an 8-bit str.

errors - How to handle Unicode errors; see help(str.decode)

for the options. Default is ‘strict’.

touch()

Set the access/modified times of this file to the current time. Create the file if it does not exist.

unlink()

utime(times)

Set the access and modified times of this file.

walk() → iterator over files and subdirs, recursively.

The iterator yields path objects naming each child item of this directory and its descendants. This requires that D.isdir().

This performs a depth-first traversal of the directory tree. Each directory is returned just before all its children.

walkdirs() → iterator over subdirs, recursively.

With the optional ‘pattern’ argument, this yields only directories whose names match the given pattern. For example, mydir.walkdirs(‘*test’) yields only directories with names ending in ‘test’.

walkfiles() → iterator over files in D, recursively.

The optional argument, pattern, limits the results to files with names that match the pattern. For example, mydir.walkfiles(‘*.tmp’) yields only files with the .tmp extension.

common.util.targetplatform module

common.util.targetplatform.targetplatform()

common.util.types module

Special types for the pipeline

common.util.types.add_metaclass(metaclass)

Class decorator for creating a class with a metaclass.

class common.util.types.long

Bases: int

This class only exists to simplify initial Python 3 support.

common.util.types.oracle2datetime(ora_date)

class common.util.types.single_float

Bases: float

A type for distinguishing single precision float from double precision floats.

class common.util.types.unicode

Bases: str

This class only exists to simplify initial Python 3 support.

common.util.utilities module

This is a module for lots of utility functions.

common.util.utilities.absdate_to_datetime(absdate, abstime=0.0)

Return a datetime object for the date described by the given absdate (the number of days since January 1, 0000, an int or float), and optionally the abstime (number of seconds).

common.util.utilities.add_days(datetime_obj, n)

Add n (float) days to the given date

common.util.utilities.angular_separation(ra0, dec0, ra1, dec1)

Calculates the angular separation between two points using the haversine formula. Input and output in radians.

common.util.utilities.attrsorted(objects, *args, **kwargs)

Return a sorted list of objects based on a given set of property names. If multiple property names are given, sort first on the first property name, and where multiple objects exist with the same first property name, sort by the second propertyname, and so on.

objects: a list, tuple, or other iterable composed of objects

with named persitent properties

reverse: reverse the sense of the sorting

For example:

attrsorted(objects, ‘property.subproperty’)

or

attrsorted(‘prop1’, ‘prop2.subprop.subsubprop’)

General usage:

• objects is an iterable of objects with persistent properties that can be sorted using rich comparison operators (>, <, >=, etc.)

attrsorted(objects, ‘property’)

or

attrsorted(objects, ‘prop1’, ‘prop2.subprop.subsubprop’)

common.util.utilities.capture_key_press(figure)

common.util.utilities.compress_strings(strings=[], separator='_')

Compress a list of strings retaining the first string followed by the unique portion of the remaining strings separated by separator.

strings: a list of strings

separator: a string to separate the elements of the compressed

string

common.util.utilities.convert_coords(coords, mode='dms')

Simple conversion between degrees and sexagesimal notation for celestial coordinates (dms = degrees:minutes:seconds, hms = hours:minutes:seconds).

WARNING: When Dec is slightly negative (e.g., -00 01 00), use a

string, or cast -0 as a float (i.e., ‘-00:01:00’ or (-0.0, 1, 0), but not -0 or -00). If you don’t, the parser will not see it as negative.

common.util.utilities.create_array(object=None, dtype=None, copy=False, order=None, subok=True, ndmin=0, shape=None)

Return an array object using the “array” factory function of the numeric version available (NumPy or NumArray), defaulting to NumPy. The syntax replicates NumPy syntax (NumArray.array.savespace is ignored). All but the last parameter descriptions are from numpy.array docstring.

object: an array, any object exposing the array interface, any

object whose __array__ method returns an array, or any (nested) sequence.

dtype: The desired data-type for the array. If not given, then

the type will be determined as the minimum type required to hold the objects in the sequence. This argument can only be used to ‘upcast’ the array. For downcasting, use the .astype(t) method.

copy: If true, then force a copy. Otherwise a copy will only occur

if __array__ returns a copy, obj is a nested sequence, or a copy is needed to satisfy any of the other requirements

order: Specify the order of the array. If order is ‘C’, then the

array will be in C-contiguous order (last-index varies the fastest). If order is ‘FORTRAN’, then the returned array will be in Fortran-contiguous order (first-index varies the fastest). If order is None, then the returned array may be in either C-, or Fortran-contiguous order or even discontiguous.

subok: If True, then sub-classes will be passed-through, otherwise

the returned array will be forced to be a base-class array

ndmin: Specifies the minimum number of dimensions that the resulting

array should have. 1’s will be pre-pended to the shape as needed to meet this requirement.

shape: shape of output array (MUST match data size)

common.util.utilities.datetime_to_absdate(datetime_obj)

Get the equivalent of the “absdate” attribute of mx.DateTime objects, for the given Python datetime object.

The absdate is the number of days (a float) since January 1, 0000.

This method returns the integral number of days since that date.

common.util.utilities.datetime_to_abstime(datetime_obj)

Get the equivalent of the “abstime” attribute of mx.DateTime objects, for the given Python datetime object.

The absdate is the number of days (a float) since January 1, 0000.

This method returns the fractional number of seconds since that date. I.e. this is minus the integral number of days since that date.

common.util.utilities.datetime_to_midnight(datetime_obj)

Returns the datetime object of the middle of the night for the given datetime object (both specified in local time).

common.util.utilities.datetime_to_mjd(datetime_obj)

Get the Modified Julian Day (mjd) for the given Python datetime object.

The mjd is the number of days (a float) since November 17, 1858.

common.util.utilities.days_in_year(year)

Returns the number of days in the given year

common.util.utilities.deg2dms(deg)

Simple conversion from degrees to dms (degrees:minutes:seconds).

common.util.utilities.deg2hms(deg)

Simple conversion from degrees to hms (hours:minutes:seconds).

common.util.utilities.dms2deg(dms)

Simple conversion from dms (degrees:minutes:seconds) to degrees.

WARNING: When Dec is slightly negative (e.g., -00 01 00), use a

string, or cast -0 as a float (i.e., ‘-00:01:00’ or (-0.0, 1, 0), but not -0 or -00). If you don’t, the parser will not see it as negative.

common.util.utilities.fold_string(string, num=80, char='', newline='\n')

Fold the given string string at num characters if len(string) > num. If char is specified, prefer splitting after nearest previous occurrance of char. The newline character can also be specified.

string: any string to be split

num: number of characters to split at

char: prefer to split after specific character in string

newline: newline character

common.util.utilities.follow_link(filename)

Return the pathname to which a symbolic link points

filename: filename of a symbolic link

common.util.utilities.get_direction_to_position(ra1, dec1, ra2, dec2)

common.util.utilities.get_distinct_entities_from(parameter_list)

This function returns the distinct values from an input list.

common.util.utilities.get_fractional_lunar_illumination(dt)

Calculates the fractional lunar illumination (FLI) based on the Moon phase. The fli is about 10% off compared to Ephem. This is due to factors that are not taken into account here, such as the distance to the Moon etc.

common.util.utilities.get_moon_phase(year, month, day)

Return moonphase (0-29), 0=new Moon, given a date. Based on http://www.daniweb.com/software-development/python/code/216727

common.util.utilities.get_moon_phase_fractional(dt)

Calculates the Moon phase:

0 = new moon 1 = full moon

This function is more accurate than get_moon_phase() and might be used to replace it.

common.util.utilities.get_tmpbase(suffix='', prefix='tmp', dir=None)

Return a basename for a temporary filename.

Loosely follows tempfile.mktemp syntax, but constructs a name based on the current time (seconds since 00:00:00 1970-01-01 UTC) with microsecond precision followed by a 6 character random string. This should virtually guarantee temp-name uniqueness. The returned string has the following form:

‘<dir>/<prefix><seconds.microseconds>.<random_string>.<suffix>’

If dir is not None, it will be supplied with a trailing ‘/’ if one is not given. If suffix is not ‘’, it will be preceeded by a ‘.’ if one is not given.

suffix: suffix to add prefix: prefix to add

dir: pathname to prepend to file (None means the same as ‘’)

NOTE: If dir is given, it will be checked for, and an Exception raised

if it does not exist.

common.util.utilities.group_list(alist, group_size)

Returns a list of lists, which together contain all the items in “alist” where all but the last list have length equal to “group_size”.

common.util.utilities.hms2deg(hms)

Simple conversion from hms (hours:minutes:seconds) to degrees.

common.util.utilities.list_to_ranges(alist)

Changes a list of numbers into a list of ranges with an increment of 1. A range is a tuple of two values, the first the starting value, the second the final value of a range.

i.e. list_to_ranges([7, 8, 9, 1, -5, -7, 0, -6, 13]) results in [(-7, -5), (0, 1), (7, 9), (13,13)]

common.util.utilities.mjd_longitude_to_last(mjd, longitude)

Estimate the Local Apparent Sidereal Time (in seconds) based on Modified Julian Date and longitude.

common.util.utilities.mjd_longitude_to_lmst(mjd, longitude)

Estimate the Local Mean Sidereal Time (in seconds) based on Modified Julian Date and longitude.

common.util.utilities.mjd_to_datetime(mjd)

Get the Modified Julian Day (mjd) for the given Python datetime object.

The mjd is the number of days (a float) since November 17, 1858.

common.util.utilities.mjd_to_gast(mjd)

Estimate the Greenwich Apparent Sidereal Time (in seconds) based on Modified Julian Date.

common.util.utilities.mjd_to_gmst(mjd)

Estimate the Greenwich Mean Sidereal Time (in seconds) based on Modified Julian Date.

common.util.utilities.myglob(filespec, sorted=True, iterator=False)

Return an optionally sorted list (or unsorted iterator) of globbed filenames

filespec: any path/filename pattern including wildcards (* or ?)

sorted: sort the filename list if not an iterator

iterator: return an iterator instead of a list (sorted is

ignored)

common.util.utilities.mysort(*args)

Return a function to sort a list of objects based on a given set of properties. If multiple properties are given, sort first on the first property, and where multiple objects exist with the same first property, sort by the second property, and so on. For example:

mysort(‘property.subproperty’)

or

mysort(‘prop1’, ‘prop2.subprop.subsubprop’)

General usage:

• objects is a list of objects with persistent properties that can be sorted using rich comparison operators (>, <, >=, etc.)

sorted(objects, key=mysort(‘property’))

or

objects.sort(key=mysort(‘prop1’, ‘prop2.subprop.subsubprop’))

NOTE: mysort() returns a key function (not a compare function),

so both sort and sorted MUST use the ‘key’ keyword.

common.util.utilities.name_insert_suffix(filename, suffix)

Example: >>> name_insert_suffix(‘myfile.fits’, ‘catalog’) myfile.catalog.fits

common.util.utilities.name_replace_suffix(filename, suffix)

Example: >>> name_insert_suffix(‘myfile.fits’, ‘cat’) myfile.cat

common.util.utilities.offset_in_arcsec(ra0, dec0, ra1, dec1)

Give the difference between two sets of coordinates in units of arcsec. This is used to calculate the arcsec to move from coordinates (ra0, dec0) to coordinates (ra1, dec1), where the positive directions are North and East.

ra0: initial RA in deg, hh:mm:ss, or (h, m, s)

dec0: initial Dec in deg, [-]dd:mm:ss, or ([-]d, m, s)

ra1: final RA in deg, hh:mm:ss, or (h, m, s)

dec1: final Dec in deg, [-]dd:mm:ss, or ([-]d, m, s)

WARNING: When Dec is slightly negative (e.g., -00 01 00), use a

string, or cast -0 as a float (i.e., ‘-00:01:00’ or (-0.0, 1, 0), but not -0 or -00). If you don’t, the parser will not see it as negative and your Dec offset will likely have the wrong sign.

common.util.utilities.open_file(filename, mode='rb')

Open a file from a (un)compressed file and return the file object. Currently supports GZip and BZip2 archives and regular files.

common.util.utilities.ra_dec_lat_lst_to_alt_az(ra, dec, lat, lst=None, ha=None, mjd=None, lon=None, dt=None)

Convert one (RA, Dec) coordinate into the corresponding (alt, az) coordinate with the help of HA (LST - RA) and latitude. HA can be supplied directly (in which case RA is redundant) or calculated. LST can be calculated with the help of MJD (or dt) and longitude if necessary. (alt, az) is returned in degrees above the horizon and degrees East of North, respectively.

ra: right ascension in degrees

dec: declination in degrees lat: latitude of the telescope in degrees lst: local sidereal time in seconds from midnight

ha: hour angle in degrees

mjd: modified Julian date lon: longitude of the telescope in degrees

dt: local observation time in UT as datetime object

common.util.utilities.sort_dict_lists(d, keys, reverse=False, length=0)

Sorts the matched lists in the input dictionary by the value in the first key, and where multiple entires exist with the same first key value, sort by the second key value, and so on. Return a copy of the input dictionary sorted as specified.

d: input dictionary with matched lists for values

keys: a list of strings representing the keys setting the

priority for the sort

reverse: reverse the order of the sort (default sorts in ascending

order)

length: return the first <length> elements

common.util.utilities.str_to_datetime(timestring)

Converts DATE/TIME string as it occurs in FITS headers to a native Python datetime object.

common.util.utilities.strcompress(input_string)

common.util.utilities.subtract_days(datetime_obj, n)

Subtract n (float) days from the given date

common.util.utilities.test_altaz()

common.util.utilities.test_fli()

context.set_project(‘ALL’) f = (ReducedScienceFrame.instrument.name == ‘OMEGACAM’)[0] f = ReducedScienceFrame(object_id=’AE82A15FA01D5B19E0407D81E60E6260’)

dt = f.DATE_OBS header = f.raw.raw_fits_data.get_header_list()[0] ra_moon = header.HIERARCH_ESO_TEL_MOON_RA.value dec_moon = header.HIERARCH_ESO_TEL_MOON_DEC.value

common.util.utilities.test_ranges()

common.util.utilities.timedelta_to_seconds(timedelta_obj)

Convert a timedelta object to seconds; returns a float

common.util.utilities.unique(parameter_list)

This function returns the distinct values from an input list.

Module contents