Distributed Processing

Definitions and Use Cases

Distributed processing means that certain recipes, like processes, can or will be executed on specially devoted processors. A recipe like process is a process which is started from the shell prompt (or via a gui) and takes only directions via arguments on the command line.

A data server is a server which can retrieve image files from the Astro-WISE image database, provide a local copy (cache) and store new images.

A database server (read Oracle server) is a server which provides the metadata for the astronomical images and the astronomical sourcelists.

A user is a person which has read and write access to (parts of) the database and therefore has a database username and password.

A visitor is only allowed to browse (parts of) the database and therefore does NOT (need to) have a database username and password.

A user can work in the Astro-WISE Environment in different ways:

  • Via one of the AW compute centres: the user does not have to install anything on his own workstation. He can only run the standard recipes and browse the database. The processes are run on specified machines of one of the AW compute centres (e.g. OmegaCEN).
  • Via a local install on one network node in a LAN. The local install should be done preferably by an expert according to the opipe installation instructions. The user will hook up to the closest database server (Groningen, Munich, Napels) and the closest data server. If the user is not in the same domain as one of the data server centers (e.g. in Leiden) a caching data server could (should) be installed. He uses the standard Python code.
  • As above, but now the user wishes to modify the standard Python code. He can do a CVS checkout of the opipe stuff and make local modifications. He does not have to install all the Python bindings since they are installed on a central node of the LAN. This is typical for experienced users or top-level software developers.
  • Via a complete local install on the user’s own workstation. The user has to install everything from Python to SWarp and should make his own Python bindings to for example eclipse and oracle. This is only for advanced software developers !

Certain processes are best performed on dedicated machines, for example

  • This task is used to create calibrated science frames. It runs best in parallel on a dedicated server (i.e. like the hpc of Groningen University).
  • This task is used to coadd a set of regridded frames. It runs best on a machine which has a lot of memory.
  • This task is used to create sourcelists from science or coadded frames. It runs best on a machine which is closely coupled to the database server.

Requirements

It is obvious that different types of users require different degrees of freedom. A user of type 1 has the least influence on his process: it is completely directed by some settings from the ‘management’ which the user cannot change.

A user of type 2 has the means to modify (and so influence) the Astro-WISE Environment by directing certain processes to specific machines, deviating from the default settings.

Users of type 3 want their modified Python code executed (on a special machine) so they should be allowed to ship their code to that special machine so that it gets executed.

Users of type 4 are on their own. Since they have modified non-Pythoncode it cannot be executed on a different node. They should always do their testing on the local node where non Python things are adapted or setup their own system of remote modified nodes.

Available tasks

The following tasks should be executed on a cluster:
ReadNoise, Bias, HotPixels, ColdPixels, DomeFlat, TwilightFlat, FringeFlat, NightSkyFlat, MasterFlat, Gain, Zeropoint, Illumination, Science, Reduce.
The following tasks could/should be executed on a separate, single node:
Coadd, SourceList, AssociateList

Implementation

User interface

From the above it is clear that users of type 1-3 at least need a web-interface to the different recipes and users of type 3-4 need also a command-line interface. Users of type 3 should be able to upload their modified code to the web-interface in order to get it executed.

The web-interface should provide an interface to all available recipes. The user simply clicks on the wanted recipe and is then prompted for the wanted input to that recipe. The user can also select on which processing machine the recipe is to be executed (can depend on type of user) with smart defaults provided of course. The user gets notified via the web-interface of the status of the job.

The web-interface might be extended with some kind of scripting-language to allow for loops etc. so if for example a user wants to make 300 sourcelists from a preselected set of coadded frames he should not have to click 300 times on recipe SourceList and supply the input. Even an interface to a (limited) version of the awe-prompt might be considered.

The user should also be able to download the products of interrest, like the coaddedregriddedframes just produced by recipe Coadd or the sourcelists produced by recipe SourceList. The output should be in FITS, VOTable or Python format.

Users of type 3-4 need also a command line interface to run a recipe (or maybe even a tool to run a recipe from the awe-prompt). A command line interface to run a recipe could look like:

Code shipping

Python code shipping can be done by zipping the whole local opipe tree and sending it to the node on which the execution should take place. Python 2.3 has the nice feature of the writepy function in the PyZipFile module. Within a few seconds the code can be packed into a zip file and shipped elsewhere, where the packed (or unzipped) zip file can be placed in the PYTHONPATH i.e.//[1.5ex]


In case a user submits his own code the context of the job is switched to MyDB. This ensures that only results from legal/accepted/committed code is allowed to polute the database.

Processing Units

A processing unit is a unique virtual processor which consists of:

  • A frontend which accepts the jobs, submits the jobs for execution, monitors the jobs and reports the results back to the clients. It also accepts the Python code from the client and uses this instead of the standard code.

    The frontend allows a process manager to reconfigure it on the fly. A frontend is designated by its network name and port, it might also have an alias.

  • A backend which executes the jobs. It can be a set of parallel nodes of a beowulf cluster or a single machine. The backend can have its own queueing system or make use of the locally implemented queueing system (i.e. as on the Groningen HPC).

The awrecipe script starts a specified recipe and decides whether it needs to split the jobs in a set op parallel jobs or one job or a combination thereof. It sends the jobs to the frontend of a processing unit and waits until all its submitted jobs have finished and creates then a logfile from all the output received from the processing units (i.e. the individual logfiles generated by all individual processes).

The precursor to the awrecipe script is the generic_runner script which starts the Cluster_Manager.

The backend which executes the jobs could be configured in the following way: It starts the Cluster_Manager (or submits a job to a queue which does the same) and waits for the Cluster_Manager to report back that it is up and ready to recieve jobs (via unique keys handed out by the backend to the Cluster_Manager). When connection has been established the backend sends the jobs to the Cluster_Manager and waits for the Cluster_Manager to report that all jobs have been executed (or not in case of errors etc.). Then the Cluster_Manager quits after sending the complete logs to the backend.

The above configuration has the big advantage that it works for locally managed beowulf clusters as well as beowulf clusters with queueing systems like the Groningen HPC. And we can use the Cluster_Manager code with a slight change in frontend to allow for contact between processor unit backend and Cluster_Manager.

Problems

Problems encountered sofar:

  • The writepy function only ships binary Python code so the configuration files which are in the opipe tree don’t get shipped. Solutions: migrate the configuration files from text to Python code or (easier) ship the configuration files as well.

  • The user specific AWE settings which also contains the context specification (i.e. in $HOME/.awe/Environment.cfg must be shipped. This could very wel be put in the same zip file.

  • Copying the shell environment variables which overrule the AWEconfiguration settings is also necessary. They could very well be stored in the same zip file.
    • The last 2 items must also be performed if the local code is NOT modified (user type 2).
    • When the user wants to use the web-interface he or she has to supply all the information via this interface, i.e. like username, context etc. (or perhaps upload his or her Environment.cfg?).
  • The recipe to be executed elsewhere can only be started from the command line since the PYTHONPATH has to be set before running the recipe. This is in fact not a problem but makes it incompatible with the current taskservers. It might be necessary to create a new server thingy.
    This problem is solved if we modify the Cluster_Manager so that it communicates with the backend of the processing unit as discussed above. Each time a job is submitted the Cluster_Manager gets started up from the shipped code and then runs the shipped code.

  • Since username/password combinations and other sensitive information are going to be ‘piped’ over the internet to remote taskservers secure connections are necessary.

  • Some parallel clusters will have a job queueing system (like the Groningen HPC). A taskserver must also be able to queue the jobs on such a system. Maybe it is possible (and allowed) to run a taskserver on the frontend machine of the new HPC.
    This problem is solved if we modify the Cluster_Manager so that it communicates with the backend of the processing unit as discussed above. Then it would suffice to have a processing unit running in our domain which submits jobs to the queue via a remote script. Note that we also need to ship the Python code of the Cluster_Manager