common.net.dpu package

Submodules

common.net.dpu.client module

client.py

exception common.net.dpu.client.DPUDpuAbort

Bases: Exception

Aborted by dpu

exception common.net.dpu.client.DPUProcAbort

Bases: Exception

Aborted by process

exception common.net.dpu.client.DPUTermAbort

Bases: Exception

Terminated

exception common.net.dpu.client.DPUTimeoutError

Bases: Exception

Timeout

exception common.net.dpu.client.DPUUserAbort

Bases: Exception

Aborted by user

common.net.dpu.client.SIGALRM_handler(sig, stack)
class common.net.dpu.client.dpu_IO(host=None, port=None, debug=0, dpu_name=None, encryptionkey=None, dbinfo=None, timeout=5, certfile='', server='', dpu_userid=None, **kw)

Bases: object

Class to provide data transfer between the server and client processes based on simple HTTP GET/POST operations.

close(**kw)
deletejobid(jobid)
deletekey(key)
exceptiontest()
getalljobids()
getalljobinfo(**kw)
getallkeys()
getcerttimeleft()
getcommand(key)
getdata(key, **kw)
getit(action, **kw)
getjob(key)
getjobid(*rest)
getkey(*rest)
getnextjob(key)
getonejob(key=None, jobnum=0)
getonelog(key=None, jobnum=0)
getrealjobid(jobid)
getrequeue(key)
getstatus(key)
getsyslogs(jobid)
getversion(*rest)
host = ''
myproxydestroy()
myproxyinit(cred_lifetime=336, proxy_lifetime=24)
port = 0
retrievejobs(key)
senddata(key, data)
sendit(action, data, **kw)
sendjob(key, data)
sendjobs(key, data)
sendjobstatus(key, data)
sendremotejobstatus(key, data)
submitjobs(key, code=True, jobs=[], environment=None, path=[], osenvironment=None)
submitremotejobs(key, zip=None, jobs=[], env=None, osenv=None)
vomsproxyinit(vo='', group='', role='', proxy_lifetime=604800)
common.net.dpu.client.dpu_get_info(dpu_name=None, items=['NAME', 'HOSTNAME', 'PORT', 'KEY', 'FQDN', 'NODES', 'PPN'])
common.net.dpu.client.dpu_get_info_as_dict()
common.net.dpu.client.dpu_key_to_str(arg)
common.net.dpu.client.dpu_packit(*args, **kwargs)
common.net.dpu.client.dpu_pwmangler(password)
common.net.dpu.client.dpu_receive(host, port, allowed_clients)
common.net.dpu.client.dpu_str_to_key(arg)
common.net.dpu.client.dpu_transmit(host, port, code)
common.net.dpu.client.dpu_unpackit(data)
common.net.dpu.client.get_dpu_info_from_string(dpu_name)
common.net.dpu.client.get_inet_socket(host, port)
common.net.dpu.client.getzippedcode(path=[])
common.net.dpu.client.unicode2string(object)

common.net.dpu.clienthandler module

clienthandler.py

class common.net.dpu.clienthandler.ClientHandler

Bases: object

This class contains all the functions which communicate with the client via the http POST protocol. class DPU inherits from this class. All method names start with ‘do_‘!

do_getalljobids(http, jobinfo, data, key=None)
do_getalljobinfo(http, jobinfo, data, key=None)
do_getallkeys(http, jobinfo, data, key=None, wantjobids=False)
do_getcommand(http, jobinfo, data, key=None)
do_getdata(http, jobinfo, data, key=None)
do_getdelkey(http, jobinfo, data, key=None)
do_getjob(http, jobinfo, data, key=None)
do_getjobid(http, jobinfo, data, key=None)
do_getkey(http, jobinfo, data, key=None, wantjobid=False)
do_getnextjob(http, jobinfo, data, key=None)
do_getonejob(http, jobinfo, data, key=None)
do_getonelog(http, jobinfo, data, key=None)
do_getrealjobid(http, jobinfo, data, key=None)
do_getrequeue(http, jobinfo, data, key=None)
do_getstatus(http, jobinfo, data, key=None)
do_getsyslogs(http, jobinfo, data, key=None)
do_getversion(http, jobinfo, data, key=None)
do_senddata(http, jobinfo, data, key=None)
do_sendjob(http, jobinfo, data, key=None)
do_sendjobs(http, jobinfo, data, key=None)
do_sendjobstatus(http, jobinfo, data, key=None)
do_sendremotejobstatus(http, jobinfo, data, key=None)
class common.net.dpu.clienthandler.GridClientHandler

Bases: common.net.dpu.clienthandler.ClientHandler

A class for the extra client functionalities needed for the Grid jobs! Thes extra methods all deal with certificates!

do_getcerttimeleft(http, jobinfo, data, key=None)
do_getmyproxydestroy(http, jobinfo, data, key=None)
do_sendmyproxyinit(http, jobinfo, data, key=None)

common.net.dpu.dpu module

common.net.dpu.general module

general.py

common.net.dpu.general.cmp_to_key(mycmp)

Convert a cmp= function into a key= function

common.net.dpu.general.gen_checkkeyfile(keyfile, passphrase=None)

This can and must be improved!!!

common.net.dpu.general.gen_checksourcecode()
common.net.dpu.general.gen_closelogfile()

gen_closelogfile closes the log file and resets stdout and stderr

common.net.dpu.general.gen_copyfile(fsrc, fdst, length=16384)

gen_copy data from file-like object fsrc to file-like object fdst

common.net.dpu.general.gen_dictcopy(jobdict)

gen_dictcopy creates a copy

common.net.dpu.general.gen_docommand(cmd, input=None)

gen_docommand excutes a shell command

common.net.dpu.general.gen_dospawn(cmd)

gen_dospawn spawns a shell command and does not wait

common.net.dpu.general.gen_getcertinfo(certfile)

This can and must be improved!!!

common.net.dpu.general.gen_getfilenames(originalfilename, filetype)

gen_getfilenames generates filenames for the wanted extensions

common.net.dpu.general.gen_getpythonversionfromcode(magic)
common.net.dpu.general.gen_getpythonversionofpythoncodeinzipfile(file)

gen_getpythonversionofpythoncodeinzipfile determines the version of the compiled python code

common.net.dpu.general.gen_getscriptbody(vardict)

gen_getscriptbody creates the script text to start the dpu_runners

common.net.dpu.general.gen_getsecretfrompath(path)
common.net.dpu.general.gen_getthreadnumber(this_thread=None)

gen_getthreadnumber generates a number per thread

common.net.dpu.general.gen_globalconfig(configdict, first=True)
common.net.dpu.general.gen_ishexbin(s)
common.net.dpu.general.gen_lineno()

gen_lineno returns the current line number in our program.

common.net.dpu.general.gen_mailto(receivers, subject, message, sender='dpuserver@wodan.sky.heaven', smtphost='localhost', replyto='noreply@localhost')
common.net.dpu.general.gen_openlogfile()

gen_openlogfile opens the log file

common.net.dpu.general.gen_readfromfile(file, length=16384)

gen_readfromfile reads a pickled object from a file and unpickles it

common.net.dpu.general.gen_resetsshtunnel(pt)

gen_resetsshtunnel finds the ssh tunnel that belongs to pid and restarts it

common.net.dpu.general.gen_shutdown()

gen_shutdown the dpu servers at a slow pace

common.net.dpu.general.gen_splitaweexepath(aweexepath)
common.net.dpu.general.gen_storeinfile(thing, dir, filename)

gen_storeinfile pickles a python object and writes it to the specfied file

common.net.dpu.general.gen_testallsshtunnels(force=False)

gen_testallsshtunnels tests whether the ssh tunnels are alive

common.net.dpu.general.gen_tolog(*args, **kwargs)

gen_tolog generates output to the screen (for debugging etc.)

common.net.dpu.general.gen_xmlspecialchars(in_string)

gen_xmlspecialchars replaces special characthers &,< and > to xml encoding

common.net.dpu.globals module

globals.py

class common.net.dpu.globals.global_struct

Bases: object

CHANGETIME = 12
CNODES = 33
CODEFILE = 0
DBPASSWORD = 19
DBUSERNAME = 18
DPUJOBIDS = 30
ENVFILE = 13
HOSTPORT = 32
Halt = <threading.Event object>
JOBDICTREV = 23
JOBDICTS = 21
JOBDICTSTATUS = 20
JOBID = 5
JOBIDS = 24
JOBIDSLIST = 25
JOBIDSREVLIST = 26
JOBINFO = 17
JOBJOBS = 11
JOBKEYS = 6
JOBLISTENTRY = 29
JOBLISTS = 28
JOBLOGS = 8
JOBMODE = 2
JOBMODI = 7
JOBSTATUS = 4
JOBTIMES = 9
JOBUSERID = 10
LASTPING = 31
LASTTIME = 14
REALJOBIDS = 38
REMOTEKEYS = 36
RUNNERSTATUS = 27
RUSERJOB = 16
SERVERNAME = 34
SERVERPORT = 35
STARTTIME = 3
STATUSTEXT = 37
SYSLOGS = 15
USERID = 1
WAITPN = 22
allsorted_dpunames = []
codefiles = {}
configoptions = {}
dpu_types = []
dpunames = {}
dpus = {}
ggv(item)
hostports = []
listeners = []
logfile = 'dpu-server.log'
logfilefd = None
logging = False
loglevel = 249
mail_address = 'helmich'
mythreads = {}
myversion = 'UNKNOWN'
otherpids = []
pidfile = 'dpu-server.pid'
port_number = 9000
pythoncodeversion = '3.4'
pythonversion = '3.4.6'
reallogfile = ''
recursivecount = 0
request_queue_size = 63
servercertfile = ''
servers = []
sgv(item, value)
smtphost = 'localhost'
sorted_dpunames = []
sshtunnelstesttime = 0.0
stderr = <common.config.startup.Myfile object>
stdout = <common.config.startup.Myfile object>
threads = []
time_last_codecheck = 0.0
time_last_mail = 0.0
version = '?'
www_name = 'Astro-Wise DistributedProcessingUnit'
www_views = ['job', 'subjob', 'mixed']
class common.net.dpu.globals.global_versiontable

Bases: dict

common.net.dpu.http module

http.py

class common.net.dpu.http.DataRequestHandler(request, client_address, server)

Bases: http.server.SimpleHTTPRequestHandler

certfile = ''
clientaddr = None
do_GET()
do_POST()
get_data(blocksize=16384, length=0)
log_message(*args, **kwargs)
log_request(*args, **kwargs)
post_data = b''
server_connect_string()
show_dump()
show_index()

Here we show the list of all possible dpu’s this server is running, together with the possibility to select any one of them for further information.

show_loglevel()
show_status(dpuname, sorting, view, refresh=60, jobid=None, jobdictviewmask=0, sortitem='J', user='all')
show_xml(dpuname, a_key=None)
tolog(*args, **kwargs)

tolog generates output to the screen (for debugging etc.)

class common.net.dpu.http.ThreadingHTTPServer(address, handler, certfile='')

Bases: socketserver.ThreadingMixIn, http.server.HTTPServer

Class ThreadingHTTPServer allows for a secure or non secure connection to a client.

allow_reuse_address = True
certfile = ''
handle_error(request, client_address)
hostport = ()
secure = False
serve_forever()
tolog(*args, **kwargs)

tolog generates output to the screen (for debugging etc.)

common.net.dpu.jobhandler module

jobhandler.py

class common.net.dpu.jobhandler.JobHandler

Bases: dict

This class contains methods for the dpu class which are NOT dependent on dpu type! The method names all start with job!

job_addjob(key, jobkey, job, info)
job_addjobs(key, jobdict)
job_cascaderunnerstatus(key, jobid)
job_cascadesubjobstatus(key, jobid)
job_cleanup(key)
job_countmissingsyslogs(key)
job_deletecodefile(key)
job_deletejob(key)
job_deletesubjob(key, jobdict_idx)
job_findfirstrealjob(key, jobid, subjob_id)
job_findnextjobtostart(key)
job_getdputime(jd, default=600)

This routine tries to get the needed DPU_TIME from the jobdict.

job_getjobdictindex(key, idx)

returns the jobdict_index to which the job with index idx belongs

job_getjobstatus(key)
job_getosenv(key)
job_getsubjobjobdictstatus(key, jobdict, idx)
job_getsubjobstatus(key, jobdict_idx=0, jidx=None, jobdict=None)
job_jobid2key(jobid)
job_key2jobid(key)
job_makecodefile(key, code)
job_makejobids(key, idx_s, idx_e, count)
job_nextjobkey(key)
job_queuedelayedjobs()
job_queuejobs(key, jobdict)
job_readcodefile(key, length=16384)
job_removewaitport(key, jobid)
job_resetjob(key)
job_restartjob(key)
job_returnjobs(key)
job_runnerabort(key, runkey)
job_runnersstop(key, jobid)
job_runnerstest(key, jobid)
job_sendremotejobstatus(key, idx)
job_setjobstatus(key)
job_setsubjobstatus(key, jobdict_idx)
job_skipjob(key, errtext)
job_skipjobdict(key, jobdict, errtext)
job_skipjobdicts(key, jobdict, errtext)
job_startjob(key)
job_startnextjob(key, jobdict_idx)
job_wakeupotherjobs(key, jobid, start_other_jobs)

common.net.dpu.main module

common.net.dpu.runner module

runner.py

common.net.dpu.runner.getrunnerfilename()
common.net.dpu.runner.getrunnermodulename()

common.net.dpu.test module

common.net.dpu.test.getresults(dpu, jobid)
common.net.dpu.test.main(*kw_list, **kw_dict)

common.net.dpu.testjob module

class common.net.dpu.testjob.testjob(**kw_dict)

Bases: object

connecttest = 0
dstest = 0
execute()
gracefulexit = False
matplotlib = True
procabort = False
raiseexception = False
showinfo = []
systemop = False
termabort = False
timeconstant = 60.0

Module contents

dpu