FAQ¶
General¶
Introductory Material¶
Q: What is Astro-WISE?
A: Astro-WISE for Astronomical Wide-field Imaging System Europe, is an E.U. consortium and associated information system dedicated to the handling of wide-field image data resulting from large astronomical surveys. Please see the main page for a complete description.
Q: What is the Astro-WISE Environment?
A: The Astro-WISE Environment, or simply AWE, is the information system built to handle this flood of astronomical data. Start here to learn more about it.
Q: What language is the Astro-WISE Environment written in?
A: The Astro-WISE Environmentis written mainly in the Python programming/scripting language. There are also Python utility modules written in C (see HOW-TO Introduction) but these are wrapped in Python to interface with the system.
Q: What kind of web-services does AWE offer?
A: See HOW-TO Introduction for a complete list of what AWE offers.
Getting Started¶
Q: I can’t access anything but the basics in the AWE, nor can I access the database. What’s wrong?
A: Do you have an AWE account? If not, please contact an Astro-WISE representative.
Q: I keep seeing references to $AWEPIPE. What is it?
A: $AWEPIPE is the environmental variable that should point to your “awe” checkout, the code-base for AWE (e.g., \(\sim\)/awe/).
Q: How do I start AWE?
A: There are many options, but the typical one is to simply use AWE on a properly configured system:
...]$ awe Python 3.5.1 (default, Nov 1 2016, 21:12:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux Type "help", "copyright", "credits" or "license" for more information. Welcome to the Astro-WISE Environment Importing Astro-WISE packages. Please wait... . . . awe>You can also temporarily modify environmental variables that AWE uses so that AWE starts with a different configuration:
...]$ env AWEPIPE=~/test-awe/ awe or ...]$ env database_engine= awe
The first example causes AWE to use a code-base located in a specified directory (\(\sim\)/test-awe/ here) and the second disables use of the database (if you need to work offline for some reason).
Documentation¶
Q: I’m a new user. Where should I look for documentation?
A: The first place any new user should look for documentation is the HOW-TOs.
Q: What are docstrings?
A: Docstrings are the inline comments added to Python code that allow for an automated documentation system called PyDoc. Docstrings are contained within triple quotes:
'''this is a docstring''' """this is also a docstring""" # this is just a comment
GitLab (Git)¶
Q: What is the address of the Astro-WISE GitLab server?
Q: What is the proper way to fully update a code-base?
A: First, make sure you are in the top-level directory (or whatever directory you want fully updated) and issue the following command:
...]$ git checkout master ...]$ git pull
Q: How do I switch between the latest $AWEPIPE version and the master version?
A: To switch to the most recent check out use
...]$ git checkout develop ...]$ git pull
and to switch back to the master version, use
...]$ git checkout master ...]$ git pull
in the ‘awe’ directory.
Data Preparation¶
Q: I am about to go observing with an AWE-supported instrument. Is there anything special that I should do?
A: Please look at HOW-TO Schedule Astro-WISE compliant observations for guidelines.
Q: I want to process the data I took on < insert favorite instrument > with AWE. What do I need to do?
A: AWE was created specifically for OmegaCAM and will handle OmegaCAM data “out-of-the-box”. There is also support for the instruments listed on the Supported Data Sources page. If your instrument is not there, contact an AWE representative for verification and have a look at HOW-TO Define a new instrument in the Astro-WISE System (to enable ingesting data from it) to get started.
Ingesting¶
Q: What is data ingestion and how do I do it?
A: In order for raw data to be processed in the Astro-WISE system, it must first be ingested, or imported into the system. Please see HOW-TO Ingest raw data into the database for a complete description.
Q: How can the data be categorized for ingestion in AWE?
A: Data in AWE is catagorized by purpose:
- bias frames designated to determine the instrument read noise
- a raw bias or zero second exposure frames
- a frame taken to measure the dark current of the instrument
- a specific series of frames designed to determine the gain of the instrument
- a raw flat frame taken from a screen mounted within the dome enclosure
- a raw flat frame taken at twilight on the sky
- any frame taken for the purpose of science (excluding photometric calibration)
- a science frame taken with the specific purpose of photometric calibration
Q: After I ingested my data, all the filenames were different. What happened?
A: The filenames were converted to AWE cannonical names of the form:
<instrument>.<date_obs>.fits or <instrument>.<date_obs>_n.fits
The first from is for multi-extension FITS images, the second is for single-extension FITS images where n is the extension number.
Dates and Times¶
Q: How are observation dates defined in AWE?
A: An observation night is based on the local date at sunset and not on UTC. See HOW-TO Work with dates and times in AWE for more information.
Q: How are dates stored in the database?
A: In UTC only.
Parallel Processing
Q: What is the parallel processing interface in AWE?
A: The DPU (Distruibuted Processing Unit) is used to run tasks on the parallel compute cluster. It is initialized when AWE starts:
...]$ awe . . . Importing Astro-WISE packages. Please wait... Initializing Distributed Processing Unit... Current profile: . . . awe>
To initiallize it manually, use this method:
awe> from astro.recipes.mods.dpu import Processor awe> my_dpu = Processor(Env['dpu_name'])
Q: How do I find the instruments, task identifiers, and options supported by the DPU’s run method: dpu.run()?
A: This information is somewhat hidden. The supported instrument list can be seen by querying the HeaderTranslatorFactory:
awe> from astro.instrument.HeaderTranslatorFactory import supported_instrument_list awe> print(supported_instrument_list)
Task identifiers can be found like this:
awe> from astro.recipes.mods import Pipeline awe> for id in Pipeline.get_available_sequence_identifiers(): print(id)
And the options can be printed in this way:
awe> from astro.recipes.util.ArgumentParser import ArgumentParser awe> for opt in ArgumentParser().opts: print(opt)
This gives you a list of tuples of the form (short_opt, long_opt, definition).
awe-prompt¶
Q: What is namespace and how can I see it?
A: Simply speaking, namespace is all modules, classes, attributes, and methods available in the current scope (i.e., level). If the module, class, attribute, or method you need is not visible in the current namespace, then it cannot be used. It is either loaded within the namespace at a different scope, or it is not there at all. The builtins dir() and help() allow you to explore the namespace.
dir() without any arguments gives the namespace of the current scope. It typically will give Python builtins and any modules pre-loaded at startup or during the current session. Calling dir() with a module, class, attribute, or method as the argument will give you the namespace at that level or an arbitrary level:
awe> dir() ['__builtin__', '__builtins__', '__doc__', '__file__', '__name__', '__version_ _', 'astro', 'atexit', 'os', 'pydoc', 'readline', 'rlcompleter', 'sys', 'users tartup'] awe> dir(os) ['EX_CANTCREAT', 'EX_CONFIG', 'EX_DATAERR', 'EX_IOERR', 'EX_NOHOST', 'EX_NOINP UT', 'EX_NOPERM', 'EX_NOUSER', 'EX_OK', 'EX_OSERR', 'EX_OSFILE', 'EX_PROTOCOL' . . . ttyname', 'umask', 'uname', 'unlink', 'unsetenv', 'utime', 'wait', 'waitpid', 'walk', 'write'] awe> dir(os.write) ['__call__', '__class__', '__cmp__', '__delattr__', '__doc__', '__getattribute __', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__' , '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__str__']
help() can be used in a similar way, but it gives all the docstrings recursively for the module, class, or method, or for the parent class of the attribute that was given as the argument.
Q: When importing something from the awe-prompt, I get strange error messages through my screen. I know that the software is okay. What is the deal?
A: Make sure that you are not importing something from within the awe directory structure. Due to some strange Python things concerning packages, this can lead to completely intractable error messages.
Q: One of the Python recipes generates an error and I can not figure out what is wrong. What should I do?
A: Sometimes problems arise because of old compiled Python files (*.pyc); there may be leftovers of Python files that no longer exist. Remove all these files and try again.
Q: What are some usefull packages and modules available at the awe-prompt?
A: See HOW-TO Use the awe-prompt
Queries¶
Q: What are queries and how do I use them?
A: Queries in AWE are queries to the database from Python with the purpose of returning some needed information. HOW-TO Query the database from Python containes a complete explaination of the use of queries in AWE.
Q: What’s the deal with the select() method?
A: The select() method is simply a convenience method to find the most recent, valid version of a given frame. Use the Python builtin help() on a particular select method to find out exactly how it makes this choice.
Process Parameters¶
Q: Which process parameters can be configured and where?
A: Generally speaking, all process parameters can be configured from the awe-prompt prior to running a make or executing a task. HOW-TO Configure process parameters gives a complete description of the various methods for process parameter configuration.
Context¶
Q:
A:
Q:
A:
Q:
A:
Q:
A:
Q:
A:
Astro-WISE Environment¶
Q: I want to run SExtractor on my images. How do I do that ?
A: There are basically three ways to run SExtractor in the system. The most direct one would be to import ‘Sextractor’ from ‘astro/external’ and run ‘Sextractor.sex(image)’. You can also set the SExtractor configuration through this interface, and modify the list of output parameters. A second way to run SExtractor would be using the ‘sex’ method of BaseFrame or children thereof. To be concrete, instantiate a ScienceFrame object and call the ‘sex’ method. The third, and in our paradigm the most correct, way to run SExtractor, is to invoke the ‘make’ method of a Catalog object. The frame you want to extract sources from is given as a dependency to the Catalog object. For the configuration of SExtractor and to modify its outputs, you need to provide the Catalog object with two additional dependencies: a SextractorConfig object, and a list of parameters. Note that this last way of creating a SExtractor catalog uses the first method. Also note, that the output of running SExtractor is in LDAC fits format.
Q:
A:
Q:
A:
Q:
A:
AW Tutorials¶
Q:
A:
Q:
A:
Q:
A:
Calibration¶
Q:
A:
Q:
A:
Q:
A:
Image Pipeline¶
Q:
A:
Q:
A:
Q:
A:
Visualization¶
Q:
A:
Q:
A:
Q:
A:
Development¶
Q:
A:
Q:
A:
Q:
A:
