common.net.dpu package¶
Submodules¶
common.net.dpu.client module¶
client.py
-
exception
common.net.dpu.client.
DPUDpuAbort
¶ Bases:
Exception
Aborted by dpu
-
exception
common.net.dpu.client.
DPUProcAbort
¶ Bases:
Exception
Aborted by process
-
exception
common.net.dpu.client.
DPUTermAbort
¶ Bases:
Exception
Terminated
-
exception
common.net.dpu.client.
DPUTimeoutError
¶ Bases:
Exception
Timeout
-
exception
common.net.dpu.client.
DPUUserAbort
¶ Bases:
Exception
Aborted by user
-
common.net.dpu.client.
SIGALRM_handler
(sig, stack)¶
-
class
common.net.dpu.client.
dpu_IO
(host=None, port=None, debug=0, dpu_name=None, encryptionkey=None, dbinfo=None, timeout=5, certfile='', server='', dpu_userid=None, **kw)¶ Bases:
object
Class to provide data transfer between the server and client processes based on simple HTTP GET/POST operations.
-
close
(**kw)¶
-
deletejobid
(jobid)¶
-
deletekey
(key)¶
-
exceptiontest
()¶
-
getalljobids
()¶
-
getalljobinfo
(**kw)¶
-
getallkeys
()¶
-
getcerttimeleft
()¶
-
getcommand
(key)¶
-
getdata
(key, **kw)¶
-
getit
(action, **kw)¶
-
getjob
(key)¶
-
getjobid
(*rest)¶
-
getkey
(*rest)¶
-
getnextjob
(key)¶
-
getonejob
(key=None, jobnum=0)¶
-
getonelog
(key=None, jobnum=0)¶
-
getrealjobid
(jobid)¶
-
getrequeue
(key)¶
-
getstatus
(key)¶
-
getsyslogs
(jobid)¶
-
getversion
(*rest)¶
-
host
= ''¶
-
myproxydestroy
()¶
-
myproxyinit
(cred_lifetime=336, proxy_lifetime=24)¶
-
port
= 0¶
-
retrievejobs
(key)¶
-
senddata
(key, data)¶
-
sendit
(action, data, **kw)¶
-
sendjob
(key, data)¶
-
sendjobs
(key, data)¶
-
sendjobstatus
(key, data)¶
-
sendremotejobstatus
(key, data)¶
-
submitjobs
(key, code=True, jobs=[], environment=None, path=[], osenvironment=None)¶
-
submitremotejobs
(key, zip=None, jobs=[], env=None, osenv=None)¶
-
vomsproxyinit
(vo='', group='', role='', proxy_lifetime=604800)¶
-
-
common.net.dpu.client.
dpu_get_info
(dpu_name=None, items=['NAME', 'HOSTNAME', 'PORT', 'KEY', 'FQDN', 'NODES', 'PPN'])¶
-
common.net.dpu.client.
dpu_get_info_as_dict
()¶
-
common.net.dpu.client.
dpu_key_to_str
(arg)¶
-
common.net.dpu.client.
dpu_packit
(*args, **kwargs)¶
-
common.net.dpu.client.
dpu_pwmangler
(password)¶
-
common.net.dpu.client.
dpu_receive
(host, port, allowed_clients)¶
-
common.net.dpu.client.
dpu_str_to_key
(arg)¶
-
common.net.dpu.client.
dpu_transmit
(host, port, code)¶
-
common.net.dpu.client.
dpu_unpackit
(data)¶
-
common.net.dpu.client.
get_dpu_info_from_string
(dpu_name)¶
-
common.net.dpu.client.
get_inet_socket
(host, port)¶
-
common.net.dpu.client.
getzippedcode
(path=[])¶
-
common.net.dpu.client.
unicode2string
(object)¶
common.net.dpu.clienthandler module¶
clienthandler.py
-
class
common.net.dpu.clienthandler.
ClientHandler
¶ Bases:
object
This class contains all the functions which communicate with the client via the http POST protocol. class DPU inherits from this class. All method names start with ‘do_’!
-
do_getalljobids
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getalljobinfo
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getallkeys
(http, jobinfo, data, key=None, requestjob=None, wantjobids=False)¶
-
do_getcommand
(http, jobinfo, data, key=None)¶
-
do_getdata
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getdelkey
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getjob
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getjobid
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getkey
(http, jobinfo, data, key=None, requestjob=None, wantjobid=False)¶
-
do_getnextjob
(http, jobinfo, data, key=None)¶
-
do_getonejob
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getonelog
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getrealjobid
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getrequeue
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getstatus
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getsyslogs
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getversion
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_senddata
(http, jobinfo, data, key=None)¶
-
do_sendjob
(http, jobinfo, data, key=None)¶
-
do_sendjobs
(http, jobinfo, data, key=None)¶
-
do_sendjobstatus
(http, jobinfo, data, key=None)¶
-
do_sendremotejobstatus
(http, jobinfo, data, key=None)¶
-
writetoclient
(client, results)¶
-
-
class
common.net.dpu.clienthandler.
GridClientHandler
¶ Bases:
common.net.dpu.clienthandler.ClientHandler
A class for the extra client functionalities needed for the Grid jobs! Thes extra methods all deal with certificates!
-
do_getcerttimeleft
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_getmyproxydestroy
(http, jobinfo, data, key=None, requestjob=None)¶
-
do_sendmyproxyinit
(http, jobinfo, data, key=None)¶
-
common.net.dpu.dpu module¶
dpu.py
-
common.net.dpu.dpu.
ClassFactory
(params)¶
-
class
common.net.dpu.dpu.
DPU
(params)¶ Bases:
common.net.dpu.clienthandler.ClientHandler
,common.net.dpu.jobhandler.JobHandler
-
allowedfunctions
= ['GETALLJOBIDS', 'GETALLJOBINFO', 'GETALLKEYS', 'GETCOMMAND', 'GETJOB', 'GETNEXTJOB', 'GETVERSION', 'SENDJOB', 'SENDJOBSTATUS']¶
-
configoptions_general
= {'aes': <class 'str'>, 'allowedusers': <class 'str'>, 'aweexe': <class 'str'>, 'blockedusers': <class 'str'>, 'dir': <class 'str'>, 'loglevel': <class 'int'>, 'rsh': <class 'str'>, 'sh': <class 'str'>, 'swd': <class 'str'>, 'type': <class 'str'>, 'wwwlogo': <class 'str'>, 'wwwname': <class 'str'>}¶
-
dpu_canceljob
(key, sys_jobfile, sys_jobid, jobdict_idx, remote_usert)¶
-
dpu_checkmainloopthread
(checktime=None)¶
-
dpu_cleanupfiles
(file, remote_usert)¶
-
dpu_cleanupjobs
()¶
-
dpu_cleanuprequests
()¶
-
dpu_closed
()¶
-
dpu_cpfromremote
(file, rfile=None, remote_usert=())¶
-
dpu_cptoremote
(file, rfile=None, remote_usert=())¶
-
dpu_delsshtunnels
()¶
-
dpu_doremote
(input, remote_usert=())¶
-
dpu_dumpserverstructs
(force=False)¶
-
dpu_existsdirremote
(rdir, remote_usert=())¶
-
dpu_existsremote
(file, rfile=None, remote_usert=())¶
-
dpu_getaweexe
(aweversion, awetarget, awepv, awe_exes=None)¶
-
dpu_getrequestkey
()¶
-
dpu_getsshcmd
(remote_usert=(), mode='ssh', infile=None, outfile=None, command='')¶
-
dpu_getuniqueserverid
()¶
-
dpu_getwaitcommand
(key, jobid, localhost)¶
-
dpu_globalinitlock
()¶
-
dpu_globallock
(key=None)¶
-
dpu_globalunlock
(key=None)¶
-
dpu_globalunlockiflocked
()¶
-
dpu_handleclient
(http, data=b'')¶
-
dpu_jobfinish
(key, runkey)¶
-
dpu_keyinitlock
(key)¶
-
dpu_keylock
(key)¶
-
dpu_keyremovelock
(key)¶
-
dpu_keyunlock
(key)¶
-
dpu_killchildren
()¶
-
dpu_localqueuejobs
(key, jobdict, force=False)¶
-
dpu_mainloop
()¶
-
dpu_mainloopfinish
()¶
-
dpu_mainloopkey
(key)¶
-
dpu_mainloopstart
()¶
-
dpu_makedefaultjobdict
(jobs)¶
-
dpu_mkdirremote
(rdir, remote_usert=())¶
-
dpu_remotequeuejobs
(key, jobdict)¶
-
dpu_restoreserverstructs
()¶
-
dpu_rmdirremote
(rdir, remote_usert=())¶
-
dpu_rmremote
(file, rfile=None, remote_usert=())¶
-
dpu_shutdown
(reboot=False)¶
-
dpu_startmainloopthread
()¶
-
dpu_startup
(reboot=False)¶
-
dpu_storeserverchildpids
()¶
-
dpu_storeserverinfo
()¶
-
dpu_testsshtunnel
(remote_usert=(), server=False)¶
-
dpu_testsshtunnels
()¶
-
dpu_timestamp
()¶
-
dumpings
= []¶
-
dumpingscount
= 0¶
-
dumpparams
()¶
-
goon
= True¶
-
lastcheck
= 0.0¶
-
locked
= True¶
-
params
= None¶
-
requestfunctions
= ['do_getversion', 'do_getjobid', 'do_getkey', 'do_getalljobinfo', 'do_getdata', 'do_getjob', 'do_getonejob', 'do_getonelog', 'do_getstatus', 'do_getrealjobid', 'do_getsyslogs', 'do_getalljobids', 'do_getallkeys', 'do_getdelkey', 'do_getrequeue']¶
-
requests
= {}¶
-
threadcount
= 0¶
-
threadname
= ''¶
-
tolog
(*args, **kwargs)¶ tolog generates output to the screen (for debugging etc.)
-
-
class
common.net.dpu.dpu.
DPU_params
(params=None)¶ Bases:
object
-
aes
= None¶
-
allowedusers
= []¶
-
awe_executable
= '/opt/awehome/Linux-centos-7.4-x86_64/master/astro/bin/awe'¶
-
awe_exes
= []¶
-
batch_system
= 'PBS'¶
-
blockedusers
= []¶
-
changetimestamp
= 1551674575.873266¶
-
cluster_name
= None¶
-
cluster_port
= 0¶
-
cnodes
= []¶
-
constraints
= ''¶
-
default_vo
= ''¶
-
delayed_jobs
= {}¶
-
dpuproxy
= False¶
-
dread_lock
= None¶
-
dry_run
= False¶
-
dumpfile
= ''¶
-
dumptimestamp
= 1551674575.873266¶
-
excludenodes
= ''¶
-
executefile
= ''¶
-
fillnodes
= 100.0¶
-
firewall
= False¶
-
grid
= False¶
-
infofile
= ''¶
-
job_max_age_ERROR
= 3600¶
-
job_max_age_FINISHED
= 3600¶
-
job_max_age_GETDATA
= 21600¶
-
job_max_age_GETKEY
= 3600¶
-
job_max_age_SENDDATA
= 604800¶
-
job_max_age_STATUS
= 300¶
-
job_max_age_TOTAL
= 2592000¶
-
job_timeout
= 60¶
-
jobcount
= 0¶
-
jobhead
= ''¶
-
jobid2key
= {}¶
-
key_lock
= {}¶
-
lock_count
= {}¶
-
lock_start
= 0.0¶
-
loglevel
= 255¶
-
mpnode
= 0¶
-
myproxyserver
= ''¶
-
name
= ''¶
-
nodelist
= ''¶
-
paramsdumpfile
= ''¶
-
path
= ''¶
-
pkgname
= 'astro'¶
-
pkgurl
= 'http://drive.astro-wise.org:8000/grid-install'¶
-
ppnode
= 1¶
-
preferred_grid_ce
= ''¶
-
prerun
= ''¶
-
remote_dir
= ''¶
-
remote_path
= ''¶
-
remote_user
= None¶
-
removejobidfile
= ''¶
-
requeuejobidfile
= ''¶
-
resetfile
= ''¶
-
resetjobidfile
= ''¶
-
rshell
= 'ssh -n -o StrictHostKeyChecking=no'¶
-
runnertimeout
= None¶
-
scratch
= '/scratch'¶
-
secret_key
= None¶
-
secret_key_string
= ''¶
-
server_name
= None¶
-
servertunnel
= ''¶
-
servertunneluserhost
= ''¶
-
sh
= '/bin/sh'¶
-
sq_mpnode
= []¶
-
sq_nnodes
= []¶
-
sq_ppnode
= []¶
-
sq_qnames
= []¶
-
sshtunnels
= {}¶
-
swd
= ''¶
-
tport
= 0¶
-
tunnel_lock
= None¶
-
tunnel_recursion
= 0¶
-
unique_server_id
= ''¶
-
www_logo
= ''¶
-
www_name
= ''¶
-
-
class
common.net.dpu.dpu.
GridDPU
(params)¶ Bases:
common.net.dpu.dpu.DPU
-
configoptions_specific
= {'myproxyserver': <class 'str'>, 'path': <class 'str'>, 'pce': <class 'str'>, 'pkgname': <class 'str'>, 'pkgurl': <class 'str'>, 'ppn': <class 'int'>, 'remoteuhd': <class 'str'>, 'tport': <class 'int'>, 'vo': <class 'str'>}¶
-
dpu_canceljob
(key, sys_jobfile, sys_jobid, jobdict_idx, remote_usert)¶
-
dpu_certinfo
(certfile)¶
-
dpu_cleanupfiles
(file, remote_usert)¶
-
dpu_jobcancel
(key, vomsfile, grid_jobid, remote_usert=())¶
-
dpu_jobgetoutput
(key)¶
-
dpu_jobgetoutputhelper
(key, jobdict_idx, vomsfile, efile, ofile, grid_jobid)¶
-
dpu_jobstatus
(vomsfile, grid_jobid, remote_usert=())¶
-
dpu_jobsubmit
(vomsfile, vo, outfile, jdlfile, jobfile, remote_usert=())¶
-
dpu_localqueuejobs
(key, jobdict, force=False)¶
-
dpu_mainloopfinish
()¶
-
dpu_mainloopkey
(key)¶
-
dpu_myproxydestroy
(username, password, remote_usert=())¶
-
dpu_myproxygetdelegation
(username, password, remote_usert=())¶
-
dpu_myproxyinfo
(username, password, remote_usert=())¶
-
dpu_myproxyinit
(dbusername, dbpassword, certfile, keyfile, passwd, cred_lifetime=336, proxy_lifetime=24, remote_usert=())¶
-
dpu_setgridsubjobstatus
(key, jobdict_idx, jstatus)¶
-
dpu_vomsproxyinfo
(vomsfile, remote_usert=())¶
-
dpu_vomsproxyinit
(key, vomsfile='', vo='', group='', role='', proxy_lifetime=86400, remote_usert=())¶
-
dumpings
= ['myproxy_dict', 'vomsfile_dict']¶
-
myproxy_dict
= {}¶
-
vomsfile_dict
= {}¶
-
-
class
common.net.dpu.dpu.
JbomDPU
(params)¶ Bases:
common.net.dpu.dpu.DPU
-
configoptions_specific
= {'firewall': <function log at 0x7fba8209a950>, 'mpn': <class 'int'>, 'nodes': <class 'str'>, 'ppn': <class 'int'>, 'prerun': <class 'str'>, 'remoteuhd': <class 'str'>, 'servertunnel': <class 'str'>}¶
-
dpu_canceljob
(key, sys_jobfile, sys_jobid, jobdict_idx, remote_usert)¶
-
dpu_findfreenodes
()¶
-
dpu_freenode
(key, jobid, schedule)¶
-
dpu_jobfinish
(key, runkey)¶
-
dpu_localqueuejobs
(key, jobdict, force=False)¶
-
dpu_mainloopstart
()¶
-
dpu_schedulejobs
()¶
-
-
class
common.net.dpu.dpu.
PBSDPU
(params)¶ Bases:
common.net.dpu.dpu.DPU
-
configoptions_specific
= {'batch': <class 'str'>, 'firewall': <function log at 0x7fba8209a950>, 'jobhead': <class 'str'>, 'path': <class 'str'>, 'prerun': <class 'str'>, 'queue': <class 'str'>, 'remoteuhd': <class 'str'>, 'servertunnel': <class 'str'>, 'tport': <class 'int'>}¶
-
dpu_canceljob
(key, sys_jobfile, sys_jobid, jobdict_idx, remote_usert)¶
-
dpu_localqueuejobs
(key, jobdict, force=False)¶
-
-
class
common.net.dpu.dpu.
RequestThread
(requestkey, function, http, jobinfo, data, key=None)¶ Bases:
threading.Thread
-
endtime
= None¶
-
results
= None¶
-
run
()¶
-
starttime
= None¶
-
-
class
common.net.dpu.dpu.
SlurmDPU
(params)¶ Bases:
common.net.dpu.dpu.DPU
-
configoptions_specific
= {'constraints': <class 'str'>, 'dpuproxy': <function log at 0x7fba8209a950>, 'excludenodes': <class 'str'>, 'fillnodes': <class 'float'>, 'firewall': <function log at 0x7fba8209a950>, 'nodelist': <class 'str'>, 'path': <class 'str'>, 'prerun': <class 'str'>, 'queue': <class 'str'>, 'remoteuhd': <class 'str'>, 'runnertimeout': <class 'str'>, 'servertunnel': <class 'str'>, 'tport': <class 'int'>}¶
-
dpu_canceljob
(key, sys_jobfile, sys_jobid, jobdict_idx, remote_usert)¶
-
dpu_localqueuejobs
(key, jobdict, force=False)¶
-
-
common.net.dpu.dpu.
initconfig
()¶ initconfig detects the configurable parameters for all different dpu types
-
common.net.dpu.dpu.
loadconfig
(configfile)¶
-
common.net.dpu.dpu.
log
(arg)¶ log translates text into a boolean.
-
common.net.dpu.dpu.
makeparamsfromconfig
(params, configdict, start=False, new=True)¶
-
common.net.dpu.dpu.
startdpusfromconfig
(configdict, start=True, dry_run=False)¶
common.net.dpu.general module¶
general.py
-
common.net.dpu.general.
cmp_to_key
(mycmp)¶ Convert a cmp= function into a key= function
-
common.net.dpu.general.
gen_checkkeyfile
(keyfile, passphrase=None)¶ This can and must be improved!!!
-
common.net.dpu.general.
gen_checksourcecode
()¶
-
common.net.dpu.general.
gen_closelogfile
()¶ gen_closelogfile closes the log file and resets stdout and stderr
-
common.net.dpu.general.
gen_copyfile
(fsrc, fdst, length=1048576)¶ gen_copy data from file-like object fsrc to file-like object fdst
-
common.net.dpu.general.
gen_dictcopy
(jobdict)¶ gen_dictcopy creates a copy
-
common.net.dpu.general.
gen_docommand
(cmd, input=None)¶ gen_docommand excutes a shell command
-
common.net.dpu.general.
gen_dospawn
(cmd)¶ gen_dospawn spawns a shell command and does not wait
-
common.net.dpu.general.
gen_getcertinfo
(certfile)¶ This can and must be improved!!!
-
common.net.dpu.general.
gen_getfilenames
(originalfilename, filetype)¶ gen_getfilenames generates filenames for the wanted extensions
-
common.net.dpu.general.
gen_getpythonversionfromcode
(magic)¶
-
common.net.dpu.general.
gen_getpythonversionofpythoncodeinzipfile
(file)¶ gen_getpythonversionofpythoncodeinzipfile determines the version of the compiled python code
-
common.net.dpu.general.
gen_getscriptbody
(vardict)¶ gen_getscriptbody creates the script text to start the dpu_runners
-
common.net.dpu.general.
gen_getsecretfrompath
(path)¶
-
common.net.dpu.general.
gen_getthreadnumber
(this_thread=None)¶ gen_getthreadnumber generates a number per thread
-
common.net.dpu.general.
gen_globalconfig
(configdict, first=True)¶
-
common.net.dpu.general.
gen_ishexbin
(s)¶
-
common.net.dpu.general.
gen_lineno
()¶ gen_lineno returns the current line number in our program.
-
common.net.dpu.general.
gen_mailto
(receivers, subject, message, sender='dpuserver@runner-ydNR8Czk-project-121-concurrent-0', smtphost='localhost', replyto='noreply@localhost')¶
-
common.net.dpu.general.
gen_openlogfile
()¶ gen_openlogfile opens the log file
-
common.net.dpu.general.
gen_readfromfile
(file, length=1048576)¶ gen_readfromfile reads a pickled object from a file and unpickles it
-
common.net.dpu.general.
gen_resetsshtunnel
(pt)¶ gen_resetsshtunnel finds the ssh tunnel that belongs to pid and restarts it
-
common.net.dpu.general.
gen_shutdown
()¶ gen_shutdown the dpu servers at a slow pace
-
common.net.dpu.general.
gen_splitaweexepath
(aweexepath)¶
-
common.net.dpu.general.
gen_storeinfile
(thing, dir, filename)¶ gen_storeinfile pickles a python object and writes it to the specfied file
-
common.net.dpu.general.
gen_testallsshtunnels
(force=False)¶ gen_testallsshtunnels tests whether the ssh tunnels are alive
-
common.net.dpu.general.
gen_tolog
(*args, **kwargs)¶ gen_tolog generates output to the screen (for debugging etc.)
-
common.net.dpu.general.
gen_writetoclient
(client, results, buffersize=16384)¶
-
common.net.dpu.general.
gen_xmlspecialchars
(in_string)¶ gen_xmlspecialchars replaces special characthers &,< and > to xml encoding
common.net.dpu.globals module¶
globals.py
-
class
common.net.dpu.globals.
global_struct
¶ Bases:
object
-
CHANGETIME
= 12¶
-
CNODES
= 33¶
-
CODEFILE
= 0¶
-
DBPASSWORD
= 19¶
-
DBUSERNAME
= 18¶
-
DPUJOBIDS
= 30¶
-
ENVFILE
= 13¶
-
HOSTPORT
= 32¶
-
Halt
= <threading.Event object>¶
-
JOBDICTREV
= 23¶
-
JOBDICTS
= 21¶
-
JOBDICTSTATUS
= 20¶
-
JOBID
= 5¶
-
JOBIDS
= 24¶
-
JOBIDSLIST
= 25¶
-
JOBIDSREVLIST
= 26¶
-
JOBINFO
= 17¶
-
JOBJOBS
= 11¶
-
JOBKEYS
= 6¶
-
JOBLISTENTRY
= 29¶
-
JOBLISTS
= 28¶
-
JOBLOGS
= 8¶
-
JOBMODE
= 2¶
-
JOBMODI
= 7¶
-
JOBSTATUS
= 4¶
-
JOBTIMES
= 9¶
-
JOBUSERID
= 10¶
-
LASTPING
= 31¶
-
LASTTIME
= 14¶
-
REALJOBIDS
= 38¶
-
REMOTEKEYS
= 36¶
-
RUNNERSTATUS
= 27¶
-
RUSERJOB
= 16¶
-
SERVERNAME
= 34¶
-
SERVERPORT
= 35¶
-
STARTTIME
= 3¶
-
STATUSTEXT
= 37¶
-
SYSLOGS
= 15¶
-
USERID
= 1¶
-
WAITPN
= 22¶
-
allsorted_dpunames
= []¶
-
codefiles
= {}¶
-
configoptions
= {}¶
-
dpu_types
= []¶
-
dpunames
= {}¶
-
dpus
= {}¶
-
ggv
(item)¶
-
hostports
= []¶
-
listeners
= []¶
-
logfile
= 'dpu-server.log'¶
-
logfilefd
= None¶
-
logging
= False¶
-
loglevel
= 249¶
-
mail_address
= 'root'¶
-
mythreads
= {}¶
-
myversion
= 'master'¶
-
otherpids
= []¶
-
pidfile
= 'dpu-server.pid'¶
-
port_number
= 9000¶
-
pythoncodeversion
= '3.5'¶
-
pythonversion
= '3.5.1'¶
-
reallogfile
= ''¶
-
recursivecount
= 0¶
-
request_queue_size
= 63¶
-
servercertfile
= ''¶
-
servers
= []¶
-
sgv
(item, value)¶
-
smtphost
= 'localhost'¶
-
sorted_dpunames
= []¶
-
sshtunnelstesttime
= 0.0¶
-
stderr
= <common.config.startup.Myfile object>¶
-
stdout
= <common.config.startup.Myfile object>¶
-
threads
= []¶
-
time_last_codecheck
= 0.0¶
-
time_last_mail
= 0.0¶
-
version
= '?'¶
-
www_logo
= 'http://www.astro-wise.org/pics/astrowise-logo.gif'¶
-
www_name
= 'Astro-Wise DistributedProcessingUnit'¶
-
www_views
= ['job', 'subjob', 'mixed']¶
-
-
class
common.net.dpu.globals.
global_versiontable
¶ Bases:
dict
common.net.dpu.http module¶
http.py
-
class
common.net.dpu.http.
DataRequestHandler
(request, client_address, server)¶ Bases:
http.server.SimpleHTTPRequestHandler
-
certfile
= ''¶
-
clientaddr
= None¶
-
do_GET
()¶
-
do_POST
()¶
-
get_data
(blocksize=1048576, length=0)¶
-
log_message
(*args, **kwargs)¶
-
log_request
(*args, **kwargs)¶
-
post_data
= b''¶
-
server_connect_string
()¶
-
show_dump
()¶
-
show_index
()¶ Here we show the list of all possible dpu’s this server is running, together with the possibility to select any one of them for further information.
-
show_loglevel
()¶
-
show_status
(dpuname, sorting, view, refresh=60, jobid=None, jobdictviewmask=0, sortitem='J', user='all')¶
-
show_xml
(dpuname, a_key=None)¶
-
starttime
= 0.0¶
-
tolog
(*args, **kwargs)¶ tolog generates output to the screen (for debugging etc.)
-
-
class
common.net.dpu.http.
ThreadingHTTPServer
(address, handler, certfile='')¶ Bases:
socketserver.ThreadingMixIn
,http.server.HTTPServer
Class ThreadingHTTPServer allows for a secure or non secure connection to a client.
-
allow_reuse_address
= True¶
-
certfile
= ''¶
-
handle_error
(request, client_address)¶
-
hostport
= ()¶
-
secure
= False¶
-
serve_forever
()¶
-
tolog
(*args, **kwargs)¶ tolog generates output to the screen (for debugging etc.)
-
common.net.dpu.jobhandler module¶
jobhandler.py
-
class
common.net.dpu.jobhandler.
JobHandler
¶ Bases:
dict
This class contains methods for the dpu class which are NOT dependent on dpu type! The method names all start with job!
-
job_addjob
(key, jobkey, job, info)¶
-
job_addjobs
(key, jobdict)¶
-
job_cascaderunnerstatus
(key, jobid)¶
-
job_cascadesubjobstatus
(key, jobid)¶
-
job_cleanup
(key)¶
-
job_countmissingsyslogs
(key)¶
-
job_deletecodefile
(key)¶
-
job_deletejob
(key)¶
-
job_deletesubjob
(key, jobdict_idx)¶
-
job_findfirstrealjob
(key, jobid, subjob_id)¶
-
job_findnextjobtostart
(key)¶
-
job_getdputime
(jd, default=600)¶ This routine tries to get the needed DPU_TIME from the jobdict.
-
job_getjobdictindex
(key, idx)¶ returns the jobdict_index to which the job with index idx belongs
-
job_getjobstatus
(key)¶
-
job_getosenv
(key)¶
-
job_getsubjobjobdictstatus
(key, jobdict, idx)¶
-
job_getsubjobstatus
(key, jobdict_idx=0, jidx=None, jobdict=None)¶
-
job_jobid2key
(jobid)¶
-
job_key2jobid
(key)¶
-
job_makecodefile
(key, code)¶
-
job_makejobids
(key, idx_s, idx_e, count)¶
-
job_nextjobkey
(key)¶
-
job_queuedelayedjobs
()¶
-
job_queuejobs
(key, jobdict)¶
-
job_readcodefile
(key, length=16384)¶
-
job_removewaitport
(key, jobid)¶
-
job_resetjob
(key)¶
-
job_restartjob
(key)¶
-
job_returnjobs
(key)¶
-
job_runnerabort
(key, runkey)¶
-
job_runnersstop
(key, jobid)¶
-
job_runnerstest
(key, jobid)¶
-
job_sendremotejobstatus
(key, idx)¶
-
job_setjobstatus
(key)¶
-
job_setsubjobstatus
(key, jobdict_idx)¶
-
job_skipjob
(key, errtext)¶
-
job_skipjobdict
(key, jobdict, errtext)¶
-
job_skipjobdicts
(key, jobdict, errtext)¶
-
job_startjob
(key)¶
-
job_startnextjob
(key, jobdict_idx)¶
-
job_wakeupotherjobs
(key, jobid, start_other_jobs)¶
-
common.net.dpu.main module¶
main.py
-
common.net.dpu.main.
autoflush
()¶
-
common.net.dpu.main.
main
()¶
-
common.net.dpu.main.
sigchild
(signum, frame)¶
-
common.net.dpu.main.
sigdump
(signum, frame)¶
-
common.net.dpu.main.
sighup
(signum, frame)¶
-
common.net.dpu.main.
signoff
(signum, frame)¶
-
common.net.dpu.main.
sigpipe
(signum, frame)¶
-
common.net.dpu.main.
sigreload
(signum, frame)¶
-
common.net.dpu.main.
usage
()¶
common.net.dpu.runner module¶
runner.py
-
common.net.dpu.runner.
getrunnerfilename
()¶
-
common.net.dpu.runner.
getrunnermodulename
()¶
common.net.dpu.test module¶
-
common.net.dpu.test.
getresults
(dpu, jobid)¶
-
common.net.dpu.test.
main
(*kw_list, **kw_dict)¶
common.net.dpu.testjob module¶
Module contents¶
dpu