HOW-TO Use local code for the DPU

When data is processed on an external machine (via the DPU) it is possible to do this with custom Python code. For help on processing on external machines see HOW-TO Process data in a distributed (parallel) way.

To use your local code Python during processing on an external processor (DPU) a zip file must be created with all the compiled Python code files. Compiled Pythonfiles have the extension pyc, versus py for the source files. The directory structure in the zip file must be exactly the same as in the awe checkout.

In the command line environment (awe-prompt) a zip file is created of the awe checkout when the ‘process_jobs’ method of DPU is called. This zip file is send to the DPU which does the processing. So all the Python code in the local awe checkout is send and used automatically.

To use your local code in the target processor a zip file must be created of the awe checkout. To create a zip file named my_dpu_code.zip from the awe-prompt use the following code:

import sys, zipfile
my_file = file('my_dpu_code.zip','w')
zipcode = zipfile.PyZipFile(my_file, mode='w')
for directory in ["astro", "common"] :
    try:
        zipcode.writepy(os.path.join(os.environ['AWEPIPE'], directory))
    except Exception, e:
        print 'Error zipping %s!' % (os.path.join(os.environ['AWEPIPE'], directory))
        print 'Exception: ', e

zipcode.close()

To use this zip file in the target processor upload the zip file using the ‘Upload Code’ link on the main screen and check the checkbox in front.