common.net package¶
Subpackages¶
- common.net.dpu package
- Submodules
- common.net.dpu.client module
- common.net.dpu.clienthandler module
- common.net.dpu.dpu module
- common.net.dpu.general module
- common.net.dpu.globals module
- common.net.dpu.http module
- common.net.dpu.jobhandler module
- common.net.dpu.main module
- common.net.dpu.runner module
- common.net.dpu.test module
- common.net.dpu.testjob module
- Module contents
- common.net.ds package
- Submodules
- common.net.ds.aes module
- common.net.ds.client module
- common.net.ds.db module
- common.net.ds.dsendtoendtest module
- common.net.ds.general module
- common.net.ds.globals module
- common.net.ds.handler module
- common.net.ds.main module
- common.net.ds.myfile module
- common.net.ds.server module
- common.net.ds.standalone module
- Module contents
Submodules¶
common.net.AES module¶
-
common.net.AES.
bytesToString
(bytes)¶
-
common.net.AES.
createByteArrayZeros
(howMany)¶
-
common.net.AES.
decrypt
(key, block)¶
-
common.net.AES.
encrypt
(key, block)¶
-
class
common.net.AES.
rijndael
(keyasstring, block_size=16)¶ Bases:
object
-
decrypt
(ciphertextasstring)¶
-
encrypt
(plaintextasstring)¶
-
-
class
common.net.AES.
rijndael_constants
¶ Bases:
object
-
common.net.AES.
stringToBytes
(s)¶
-
common.net.AES.
test
()¶
common.net.DataserverProtocol module¶
Class DataserverProtocol Provides an interface to retrieve and store files from/to dataserver
common.net.FileProtocol module¶
class FileProtocol
common.net.GPFSProtocol module¶
class GPFSProtocol
-
exception
common.net.GPFSProtocol.
GPFSException
¶ Bases:
Exception
-
class
common.net.GPFSProtocol.
GPFSProtocol
¶ Bases:
object
-
GPFS_MOUNT
= '/gpfs'¶
-
download_from_dataserver
(path)¶ download path from dataserver
-
get
(path)¶ retrieve path from gpfs
-
gpfs_mounted
()¶ detect if gpfs is mounted
-
put
(path)¶ put path on gpfs
-
use_dataserver
()¶ can we use a dataserver on top of gpfs ?
-
common.net.GridProtocol module¶
Class GridProtocol Provides an interface to retrieve and store files from/to the Grid
- Grid notifications can be found here :
- https://operations-portal.egi.eu/dns/
-
class
common.net.GridProtocol.
GridProtocol
¶ Bases:
object
-
check
(uri)¶ Find and return remote checksum. input :
uri is the uri of the remote file- output:
- Adler32 checksum of the file as obtained via srmls -l
-
check_directory
(uri)¶ check if the remote directory exists, if not create it
-
check_exist
(uri, user=None, vo=None, usercert=None, userkey=None, password=None)¶ Check first whether there is a valid proxy, then check file exist return -1 if not, 0 if yes
-
check_hash
(pathname, uri)¶ Check local-remote filesize and hash value input :
pathname is the local file uri is the target location- output :
- fileinfo dict will be filled with specific file information (like size, hash value)
-
delete
(uri, user=None, vo=None, usercert=None, userkey=None, password=None)¶ Check first whether there is a valid proxy, if not, do proxy_init()
-
classmethod
generate_hash_value
(filepath)¶ calculate hash value, and return as hex use & 0xffffffff to ensure positive values strip the 0X on the left, and L on the right
-
get
(uri, newfilename=None, user=None, vo=None, usercert=None, userkey=None, password=None)¶ Check first whether there is a valid proxy, if not, do proxy_init()
-
get_destination
()¶ return the destination
-
init_proxy
(vo=None, usercert=None, userkey=None, password=None)¶ Check if proxy exists and still is valid, otherwise generate proxy
-
put
(pathname, dest_uri, user=None, vo=None, usercert=None, userkey=None, password=None)¶ Check first whether there is a valid proxy, then copy file input :
pathname is the local file uri is the target location
-
set_default_destination
()¶ Set the default destination
-
set_destination
(destination)¶ set the default destination
-
-
exception
common.net.GridProtocol.
GridProtocolError
(message)¶ Bases:
common.log.Error.Error
-
common.net.GridProtocol.
construct_SURL
()¶
-
common.net.GridProtocol.
get_SAPath_filter
(SE)¶
-
common.net.GridProtocol.
get_SApath
(SEhost)¶
-
common.net.GridProtocol.
get_SE_filter
()¶
-
common.net.GridProtocol.
get_VOInfoPath_filter
(SE)¶
-
common.net.GridProtocol.
get_availableSEhost
()¶
-
common.net.GridProtocol.
get_endpoint_filter
(SE)¶
-
common.net.GridProtocol.
get_port_and_srm
(SEhost)¶
-
common.net.GridProtocol.
get_port_filter
(SE)¶
-
common.net.GridProtocol.
get_search_attribute
(search_item)¶
-
common.net.GridProtocol.
get_serviceEndpoint
(SEhost)¶
-
common.net.GridProtocol.
get_storageElement
()¶ check which storage elements are available for the given VO e.g. using ldapsearch
-
common.net.GridProtocol.
select_random_SEhost
(SEhost_list)¶ Select random a SE from the list
common.net.Jobs module¶
Job Control! A proposed setup, for further discussion!
-
class
common.net.Jobs.
Job
(processor=None, wait=True, logfile=None, mesfile=None)¶ Bases:
object
The Job class allows for Jobs to be executed locally or remote.
- processor fqdn of frontend to a remote processing facility or
- empty in case of processing locally
- wait wait for process to finish when the execute function
- is called. If wait = False, the execute returns a JobTicket to examine further development of the job.
logfile The catch all logfile into which all output is collected. mesfile The file into which the messages are written.
Example:
class myJob(Job):
- def set_parameters(self, sleep_time):
- self.status = None self.sleep_time = sleep_time
- def execute(self):
- self.status = sleep(self.sleep_time)
myjob = myJob(processor=’hpcibm1.service.rug.nl:8010’, wait = True) myJob.set_parameters(7) myJob.execute() print myJob.status
-
execute
()¶
-
job_status
= None¶
-
status
()¶
-
wait_for_job
()¶
-
class
common.net.Jobs.
JobTicket
(socket=None)¶ Bases:
object
The JobTicket class keeps track of the status of an associated Job. An instance of JobTicket is returned by the Job.execute call or by the submit_jobs function.
-
job_status
()¶ The job_status function returns JOB_FINISHED, JOB_QUEUED, JOB_RUNNING or JOB_ABORTED
-
-
common.net.Jobs.
status_jobs
(tickets)¶ status_jobs returns the status of Jobs connected to the input JobTickets.
-
common.net.Jobs.
submit_jobs
(processor=None, joblist=[], wait=True)¶ submit_jobs submits a list of jobs to processor and returns an equal number of JobTickets.
-
common.net.Jobs.
wait_for_jobs
(tickets)¶ wait_for_jobs waits for all Jobs to finish which are connected to the input JobTickets.
common.net.SFTPProtocol module¶
Class SFTPProtocol Provides an interface to retrieve and store files from/to the Grid
-
class
common.net.SFTPProtocol.
SFTPProtocol
¶ Bases:
object
-
NRECURRENT
= 10¶
-
check
(pathname, uri)¶
-
check_exist
(pathname, user=None, password=None)¶
-
check_hash
(pathname, uri)¶ Check local-remote filesize and hash value input :
pathname is the local file uri is the target location- output :
- fileinfo dict will be filled with specific file information (like size, hash value)
-
check_host
(host)¶
-
check_node
(uri, user=None, password=None)¶
-
delete
(uri, user=None, password=None)¶
-
classmethod
generate_hash_value
(filepath, blocksize=65536)¶ calculate hash value, and return as hex use & 0xffffffff to ensure positive values strip the 0X on the left, and L on the right
-
get
(uri, newfilename=None, user=None, password=None)¶
-
get_destination
()¶ return the destination
-
put
(pathname, dest_uri, user=None, password=None)¶
-
set_default_destination
()¶ Set the default destination
-
set_destination
(destination)¶ set the default destination
-
-
exception
common.net.SFTPProtocol.
SFTPProtocolError
(message)¶ Bases:
common.log.Error.Error
common.net.Storage module¶
class Storage
An abstract class
-
common.net.Storage.
get_distance
(current_location, other_location)¶
common.net.StorageFactory module¶
class StorageFactory
common.net.dataserver_client module¶
-
class
common.net.dataserver_client.
Data_IO
(host, port=None, store_host=None, store_port=None, debug=0, timeout=None, data_path='', user_id='', sleep=30.0, dstid='', certfile=None, secure=False, **kw)¶ Bases:
asyncore.dispatcher
Class to provide data transfer between the server and client processes based on simple HTTP GET/POST operations.
-
cachefile
(path)¶ cachefile requests a caching server to cache a file
path = name of the file
-
checksum
(path)¶ checksum locates a file and returns for each result the path, host, port and checksum
-
delete
(path, defaultkey, host=None, port=None)¶ delete removes a file from a server, i.e. it moves the file to the ddata directory on the servers disk. The file must exist on the contacted server, the path must be returned from a locate call.
path = name of the file as returned by locate. host = optional host name where the file resides. port = optional port number of the server where the file resides.
-
get
(path=None, fd=None, handle=None, extra=None, raise_exception=True, query=None, savepath=None, test=False)¶ get obtains the file from a remote server (see _getit)
-
getcaps
()¶ Get the capability string from server.
-
getexact
(path=None, fd=None, handle=None, extra=None, raise_exception=False, query=None, savepath=None, test=False)¶ getlocal obtains the file from a local server (see _getit)
-
getid
()¶ Get the server id.
-
getlocal
(path=None, fd=None, handle=None, extra=None, raise_exception=False, query=None, savepath=None, test=False)¶ getlocal obtains the file from a local server (see _getit)
-
getremote
(path=None, fd=None, handle=None, extra=None, raise_exception=False, query=None, savepath=None, test=False)¶ getlocal obtains the file from a local server (see _getit)
-
getstats
(mode='LOCAL')¶
-
handle_close
()¶
-
handle_connect
()¶
-
handle_error
()¶
-
handle_expt
()¶
-
handle_read
()¶
-
handle_write
()¶
-
locate
(path)¶ locate locates a file, i.e. returns the url of the server where the specified file resides. All data servers are probed for the file.
path = name of the file
-
locatefile
(path)¶ locatelocal locates a file, i.e. returns the url of the server where the specified file resides. Only the addressed dataserver is probed.
path = name of the file
-
locateit
(path, local=False, remote=False)¶
-
locatelocal
(path)¶ locatelocal locates a file, i.e. returns the url of the server where the specified file resides. Only the local dataservers are probed.
path = name of the file
-
locateremote
(path)¶ locateremote locates a file, i.e. returns the url of the server where the specified file resides. Only remote dataservers are probed.
path = name of the file
-
md5sum
(path, host=None, port=None)¶ md5sum calculates the md5 check sum a file. The file must exist on the contacted server, the path must be returned from a locate call.
path = name of the file as returned by locate. host = optional host name where the file resides. port = optional port number of the server where the file resides.
-
mirrorput
(path, port=8000, directory='')¶ mirrorput make the data server retrieve file path to its sdata directory
path = path to local file port = port local ds listens on directory = change the directory where the file should be mirrored
-
ping
()¶ See if there is a server running out there and get the server info.
-
ping2
(host=None, port=None, timeout=None)¶ See if there is a server running out there
-
put
(path, test=False)¶ put stores a file on a local dataserver
path = path to local file test = just for testing
-
readable
()¶
-
register
(path)¶ register gets a dataserver to register a file which is already in one of the dataserver directories.
path = path to the file (w.r.t. the working directory)
-
release
()¶ release gets a dataserver to release the port the server is listening on
-
reload
(force=False)¶ reload lets a dataserver rescan the directory structure
force = True forces, False doesn’t
-
show
()¶
-
size
(path, host=None, port=None)¶ size calculates the size of a file. The file must exist on the contacted server, the path must be returned from a locate call.
path = name of the file as returned by locate. host = optional host name where the file resides. port = optional port number of the server where the file resides.
-
stat
(path, host=None, port=None)¶ stat returns the stat tuple a file. The file must exist on the contacted server, the path must be returned from a locate call.
path = name of the file as returned by locate. host = optional host name where the file resides. port = optional port number of the server where the file resides.
-
takeover
(port)¶ takeover gets a dataserver to take over the port the current server is listening on
-
testcache
()¶ testcache tests whether the server is allowed to cache files
-
testfile
(path)¶ testfile tests whether a file exists on the server
path = name of the file
-
teststore
()¶ teststore tests whether the server is allowed to store files
-
writable
()¶
-
-
common.net.dataserver_client.
SIGALRM_handler
(sig, stack)¶
-
class
common.net.dataserver_client.
Storage
(host, port=None, store_host=None, store_port=None, debug=0, timeout=None, data_path='', user_id='', sleep=30.0, dstid='', certfile=None, secure=False, **kw)¶ Bases:
common.net.dataserver_client.Data_IO
-
object_to_commit
(data_object)¶
-
retrieve
(data_object)¶
-
store
(data_object)¶
-
-
exception
common.net.dataserver_client.
TimeoutError
¶ Bases:
Exception
-
common.net.dataserver_client.
dataserver_result_to_dict
(result)¶ method to make dictionary of dataserver locate result
-
common.net.dataserver_client.
fetch_file_from_dataserver
(filename, hostport=None, savepath=None, query=None)¶ method to retrieve a file from a dataserver
-
common.net.dataserver_client.
get_inet_socket
(host, port)¶
-
common.net.dataserver_client.
main
()¶
common.net.dataserver_proxy module¶
common.net.dpu_client module¶
common.net.dpu_client_test module¶
common.net.dpu_tools module¶
dpu_tools module
bundles tools for handling the dpu and its input/output
-
common.net.dpu_tools.
parse_dpu_status
(input_string)¶ parse status string DPU, return dict, example: in: “Q/R/N/E 10/2/0/0” out: {“status”:”running”, “queued”:10, “running”:2, “normal”:0, “error”:0}
common.net.favicon module¶
favicon.py contains the binary data for a webserver favicon.ico, used by the astro-wise dpu!
common.net.gpg module¶
-
common.net.gpg.
gpgdecrypt
(fin=None, lin=[], gpghomedir=None)¶ gpgdecrypt decrypts the encrypted lines read from file object fin. It returns a list of strings without newlines!
-
common.net.gpg.
gpgencrypt
(lines, defaultkey, fout=None, lout=[], gpghomedir=None)¶ gpgencrypt encrypts lines with defaultkey and writes them to file object fout. lines should NOT contain newlines!
common.net.iRODSProtocol module¶
Class iRODSProtocol Provides an interface to retrieve and store files from/to an iRODS server
-
class
common.net.iRODSProtocol.
iRODSProtocol
¶ Bases:
object
-
check
(pathname, uri)¶
-
check_exist
(pathname, user=None, password=None)¶
-
check_hash
(pathname, uri, user=None, password=None)¶ Check local-remote filesize and hash value input :
pathname is the local file uri is the target location- output :
- fileinfo dict will be filled with specific file information (like size, hash value)
-
check_node
(uri=None, user=None, password=None)¶
-
delete
(uri, newfilename=None, user=None, password=None)¶
-
classmethod
generate_hash_value
(filepath, blocksize=65536)¶ calculate hash value, and return as hex use & 0xffffffff to ensure positive values strip the 0X on the left, and L on the right
-
get
(uri, newfilename=None, user=None, password=None)¶
-
get_destination
()¶ return the destination
-
iinit
(password=None)¶
-
put
(pathname, dest_uri, user=None, password=None)¶
-
set_default_destination
()¶ Set the default destination
-
set_destination
(destination)¶ set the default destination
-
-
exception
common.net.iRODSProtocol.
iRODSProtocolError
(message)¶ Bases:
common.log.Error.Error
common.net.makezip_for_dpu module¶
Script for making a python code compiled zip file, which can be send to the dpu The python code is copied to a temp dir and compiled
Specify the target to use as -t or –target_dir, example :
awe makezip_for_dpu.py -t common awe makezip_for_dpu.py -t astro awe makezip_for_dpu.py -t awlofar
-
common.net.makezip_for_dpu.
main
()¶ main method
-
common.net.makezip_for_dpu.
visit
(arg, dirname, names)¶
common.net.support module¶
Contains support functions for servers.
-
common.net.support.
get_pathname
(path)¶ Split a url or filename (optionally including a complete path) into parts
- RETURNS:
- tuple with (head, tail, upath)
- EXAMPLE:
- print get_pathname(‘/a/b/c/d.fits’) (‘a/b/c’, ‘d.fits’, ‘a/b/c/d.fits’) print get_pathname(‘http://some.where:1234/e/f/g.fits’) (‘e/f’, ‘g.fits’, ‘e/f/g.fits’) print get_pathname(‘h.fits’) (‘’, ‘h.fits’, ‘h.fits’)
-
common.net.support.
test_path
(prefix, filename, directories)¶ Return complete path to a file, if it exists in given directories
The prefix is prefixed to each directory in directories. The directories argument contains a list of directories.
Module contents¶
Network