common.net package

Submodules

common.net.AES module

class common.net.AES.AES(key)

Bases: object

decrypt(ciphertext)
encrypt(plaintext)
makeIV(n)
common.net.AES.bytesToString(bytes)
common.net.AES.createByteArrayZeros(howMany)
common.net.AES.decrypt(key, block)
common.net.AES.encrypt(key, block)
class common.net.AES.rijndael(keyasstring, block_size=16)

Bases: object

decrypt(ciphertextasstring)
encrypt(plaintextasstring)
class common.net.AES.rijndael_constants

Bases: object

common.net.AES.stringToBytes(s)
common.net.AES.test()

common.net.DataserverProtocol module

Class DataserverProtocol Provides an interface to retrieve and store files from/to dataserver

class common.net.DataserverProtocol.DataserverProtocol

Bases: object

check_exist(filename)
get(uri)
get_destination()
put(pathname, uri)
set_destination(destination)

common.net.FileProtocol module

class FileProtocol

class common.net.FileProtocol.FileProtocol

Bases: object

get(uri)

get or retrieve the file

classmethod path_from_uri(uri)

return the path (and filename) part of the uri

put(uri)

put or store the file

common.net.GPFSProtocol module

class GPFSProtocol

exception common.net.GPFSProtocol.GPFSException

Bases: Exception

class common.net.GPFSProtocol.GPFSProtocol

Bases: object

GPFS_MOUNT = '/gpfs'
download_from_dataserver(path)

download path from dataserver

get(path)

retrieve path from gpfs

gpfs_mounted()

detect if gpfs is mounted

put(path)

put path on gpfs

use_dataserver()

can we use a dataserver on top of gpfs ?

common.net.GridProtocol module

Class GridProtocol Provides an interface to retrieve and store files from/to the Grid

Grid notifications can be found here :
https://operations-portal.egi.eu/dns/
class common.net.GridProtocol.GridProtocol

Bases: object

check(uri)

Find and return remote checksum. input :

uri is the uri of the remote file
output:
Adler32 checksum of the file as obtained via srmls -l
check_directory(uri)

check if the remote directory exists, if not create it

check_exist(uri, user=None, vo=None, usercert=None, userkey=None, password=None)

Check first whether there is a valid proxy, then check file exist return -1 if not, 0 if yes

check_hash(pathname, uri)

Check local-remote filesize and hash value input :

pathname is the local file uri is the target location
output :
fileinfo dict will be filled with specific file information (like size, hash value)
delete(uri, user=None, vo=None, usercert=None, userkey=None, password=None)

Check first whether there is a valid proxy, if not, do proxy_init()

classmethod generate_hash_value(filepath)

calculate hash value, and return as hex use & 0xffffffff to ensure positive values strip the 0X on the left, and L on the right

get(uri, newfilename=None, user=None, vo=None, usercert=None, userkey=None, password=None)

Check first whether there is a valid proxy, if not, do proxy_init()

get_destination()

return the destination

init_proxy(vo=None, usercert=None, userkey=None, password=None)

Check if proxy exists and still is valid, otherwise generate proxy

put(pathname, dest_uri, user=None, vo=None, usercert=None, userkey=None, password=None)

Check first whether there is a valid proxy, then copy file input :

pathname is the local file uri is the target location
set_default_destination()

Set the default destination

set_destination(destination)

set the default destination

exception common.net.GridProtocol.GridProtocolError(message)

Bases: common.log.Error.Error

common.net.GridProtocol.construct_SURL()
common.net.GridProtocol.get_SAPath_filter(SE)
common.net.GridProtocol.get_SApath(SEhost)
common.net.GridProtocol.get_SE_filter()
common.net.GridProtocol.get_VOInfoPath_filter(SE)
common.net.GridProtocol.get_availableSEhost()
common.net.GridProtocol.get_endpoint_filter(SE)
common.net.GridProtocol.get_port_and_srm(SEhost)
common.net.GridProtocol.get_port_filter(SE)
common.net.GridProtocol.get_search_attribute(search_item)
common.net.GridProtocol.get_serviceEndpoint(SEhost)
common.net.GridProtocol.get_storageElement()

check which storage elements are available for the given VO e.g. using ldapsearch

common.net.GridProtocol.select_random_SEhost(SEhost_list)

Select random a SE from the list

common.net.Jobs module

Job Control! A proposed setup, for further discussion!

class common.net.Jobs.Job(processor=None, wait=True, logfile=None, mesfile=None)

Bases: object

The Job class allows for Jobs to be executed locally or remote.

processor fqdn of frontend to a remote processing facility or
empty in case of processing locally
wait wait for process to finish when the execute function
is called. If wait = False, the execute returns a JobTicket to examine further development of the job.

logfile The catch all logfile into which all output is collected. mesfile The file into which the messages are written.

Example:

class myJob(Job):

def set_parameters(self, sleep_time):
self.status = None self.sleep_time = sleep_time
def execute(self):
self.status = sleep(self.sleep_time)

myjob = myJob(processor=’hpcibm1.service.rug.nl:8010’, wait = True) myJob.set_parameters(7) myJob.execute() print myJob.status

execute()
job_status = None
status()
wait_for_job()
class common.net.Jobs.JobTicket(socket=None)

Bases: object

The JobTicket class keeps track of the status of an associated Job. An instance of JobTicket is returned by the Job.execute call or by the submit_jobs function.

job_status()

The job_status function returns JOB_FINISHED, JOB_QUEUED, JOB_RUNNING or JOB_ABORTED

common.net.Jobs.status_jobs(tickets)

status_jobs returns the status of Jobs connected to the input JobTickets.

common.net.Jobs.submit_jobs(processor=None, joblist=[], wait=True)

submit_jobs submits a list of jobs to processor and returns an equal number of JobTickets.

common.net.Jobs.wait_for_jobs(tickets)

wait_for_jobs waits for all Jobs to finish which are connected to the input JobTickets.

common.net.SFTPProtocol module

Class SFTPProtocol Provides an interface to retrieve and store files from/to the Grid

class common.net.SFTPProtocol.SFTPProtocol

Bases: object

NRECURRENT = 10
check(pathname, uri)
check_exist(pathname, user=None, password=None)
check_hash(pathname, uri)

Check local-remote filesize and hash value input :

pathname is the local file uri is the target location
output :
fileinfo dict will be filled with specific file information (like size, hash value)
check_host(host)
check_node(uri, user=None, password=None)
delete(uri, user=None, password=None)
classmethod generate_hash_value(filepath, blocksize=65536)

calculate hash value, and return as hex use & 0xffffffff to ensure positive values strip the 0X on the left, and L on the right

get(uri, newfilename=None, user=None, password=None)
get_destination()

return the destination

put(pathname, dest_uri, user=None, password=None)
set_default_destination()

Set the default destination

set_destination(destination)

set the default destination

exception common.net.SFTPProtocol.SFTPProtocolError(message)

Bases: common.log.Error.Error

common.net.Storage module

class Storage

An abstract class

class common.net.Storage.Storage

Bases: object

retrieve()
store()
common.net.Storage.get_distance(current_location, other_location)

common.net.StorageFactory module

class StorageFactory

class common.net.StorageFactory.StorageFactory

Bases: object

delegate()

common.net.dataserver_client module

class common.net.dataserver_client.Data_IO(host, port=None, store_host=None, store_port=None, debug=0, timeout=None, data_path='', user_id='', sleep=30.0, dstid='', certfile=None, secure=False, **kw)

Bases: asyncore.dispatcher

Class to provide data transfer between the server and client processes based on simple HTTP GET/POST operations.

cachefile(path)

cachefile requests a caching server to cache a file

path = name of the file

checksum(path)

checksum locates a file and returns for each result the path, host, port and checksum

delete(path, defaultkey, host=None, port=None)

delete removes a file from a server, i.e. it moves the file to the ddata directory on the servers disk. The file must exist on the contacted server, the path must be returned from a locate call.

path = name of the file as returned by locate. host = optional host name where the file resides. port = optional port number of the server where the file resides.

get(path=None, fd=None, handle=None, extra=None, raise_exception=True, query=None, savepath=None, test=False)

get obtains the file from a remote server (see _getit)

getcaps()

Get the capability string from server.

getexact(path=None, fd=None, handle=None, extra=None, raise_exception=False, query=None, savepath=None, test=False)

getlocal obtains the file from a local server (see _getit)

getid()

Get the server id.

getlocal(path=None, fd=None, handle=None, extra=None, raise_exception=False, query=None, savepath=None, test=False)

getlocal obtains the file from a local server (see _getit)

getremote(path=None, fd=None, handle=None, extra=None, raise_exception=False, query=None, savepath=None, test=False)

getlocal obtains the file from a local server (see _getit)

getstats(mode='LOCAL')
handle_close()
handle_connect()
handle_error()
handle_expt()
handle_read()
handle_write()
locate(path)

locate locates a file, i.e. returns the url of the server where the specified file resides. All data servers are probed for the file.

path = name of the file

locatefile(path)

locatelocal locates a file, i.e. returns the url of the server where the specified file resides. Only the addressed dataserver is probed.

path = name of the file

locateit(path, local=False, remote=False)
locatelocal(path)

locatelocal locates a file, i.e. returns the url of the server where the specified file resides. Only the local dataservers are probed.

path = name of the file

locateremote(path)

locateremote locates a file, i.e. returns the url of the server where the specified file resides. Only remote dataservers are probed.

path = name of the file

md5sum(path, host=None, port=None)

md5sum calculates the md5 check sum a file. The file must exist on the contacted server, the path must be returned from a locate call.

path = name of the file as returned by locate. host = optional host name where the file resides. port = optional port number of the server where the file resides.

mirrorput(path, port=8000, directory='')

mirrorput make the data server retrieve file path to its sdata directory

path = path to local file port = port local ds listens on directory = change the directory where the file should be mirrored

ping()

See if there is a server running out there and get the server info.

ping2(host=None, port=None, timeout=None)

See if there is a server running out there

put(path, test=False)

put stores a file on a local dataserver

path = path to local file test = just for testing

readable()
register(path)

register gets a dataserver to register a file which is already in one of the dataserver directories.

path = path to the file (w.r.t. the working directory)

release()

release gets a dataserver to release the port the server is listening on

reload(force=False)

reload lets a dataserver rescan the directory structure

force = True forces, False doesn’t

show()
size(path, host=None, port=None)

size calculates the size of a file. The file must exist on the contacted server, the path must be returned from a locate call.

path = name of the file as returned by locate. host = optional host name where the file resides. port = optional port number of the server where the file resides.

stat(path, host=None, port=None)

stat returns the stat tuple a file. The file must exist on the contacted server, the path must be returned from a locate call.

path = name of the file as returned by locate. host = optional host name where the file resides. port = optional port number of the server where the file resides.

takeover(port)

takeover gets a dataserver to take over the port the current server is listening on

testcache()

testcache tests whether the server is allowed to cache files

testfile(path)

testfile tests whether a file exists on the server

path = name of the file

teststore()

teststore tests whether the server is allowed to store files

writable()
common.net.dataserver_client.SIGALRM_handler(sig, stack)
class common.net.dataserver_client.Storage(host, port=None, store_host=None, store_port=None, debug=0, timeout=None, data_path='', user_id='', sleep=30.0, dstid='', certfile=None, secure=False, **kw)

Bases: common.net.dataserver_client.Data_IO

object_to_commit(data_object)
retrieve(data_object)
store(data_object)
exception common.net.dataserver_client.TimeoutError

Bases: Exception

common.net.dataserver_client.dataserver_result_to_dict(result)

method to make dictionary of dataserver locate result

common.net.dataserver_client.fetch_file_from_dataserver(filename, hostport=None, savepath=None, query=None)

method to retrieve a file from a dataserver

common.net.dataserver_client.get_inet_socket(host, port)
common.net.dataserver_client.main()

common.net.dataserver_proxy module

common.net.dpu_client module

common.net.dpu_client_test module

common.net.dpu_tools module

dpu_tools module

bundles tools for handling the dpu and its input/output

common.net.dpu_tools.parse_dpu_status(input_string)

parse status string DPU, return dict, example: in: “Q/R/N/E 10/2/0/0” out: {“status”:”running”, “queued”:10, “running”:2, “normal”:0, “error”:0}

common.net.favicon module

favicon.py contains the binary data for a webserver favicon.ico, used by the astro-wise dpu!

common.net.gpg module

common.net.gpg.gpgdecrypt(fin=None, lin=[], gpghomedir=None)

gpgdecrypt decrypts the encrypted lines read from file object fin. It returns a list of strings without newlines!

common.net.gpg.gpgencrypt(lines, defaultkey, fout=None, lout=[], gpghomedir=None)

gpgencrypt encrypts lines with defaultkey and writes them to file object fout. lines should NOT contain newlines!

common.net.iRODSProtocol module

Class iRODSProtocol Provides an interface to retrieve and store files from/to an iRODS server

class common.net.iRODSProtocol.iRODSProtocol

Bases: object

check(pathname, uri)
check_exist(pathname, user=None, password=None)
check_hash(pathname, uri, user=None, password=None)

Check local-remote filesize and hash value input :

pathname is the local file uri is the target location
output :
fileinfo dict will be filled with specific file information (like size, hash value)
check_node(uri=None, user=None, password=None)
delete(uri, newfilename=None, user=None, password=None)
classmethod generate_hash_value(filepath, blocksize=65536)

calculate hash value, and return as hex use & 0xffffffff to ensure positive values strip the 0X on the left, and L on the right

get(uri, newfilename=None, user=None, password=None)
get_destination()

return the destination

iinit(password=None)
put(pathname, dest_uri, user=None, password=None)
set_default_destination()

Set the default destination

set_destination(destination)

set the default destination

exception common.net.iRODSProtocol.iRODSProtocolError(message)

Bases: common.log.Error.Error

common.net.makezip_for_dpu module

Script for making a python code compiled zip file, which can be send to the dpu The python code is copied to a temp dir and compiled

Specify the target to use as -t or –target_dir, example :

awe makezip_for_dpu.py -t common awe makezip_for_dpu.py -t astro awe makezip_for_dpu.py -t awlofar

common.net.makezip_for_dpu.main()

main method

common.net.makezip_for_dpu.visit(arg, dirname, names)

common.net.support module

Contains support functions for servers.

common.net.support.get_pathname(path)

Split a url or filename (optionally including a complete path) into parts

RETURNS:
tuple with (head, tail, upath)
EXAMPLE:
print get_pathname(‘/a/b/c/d.fits’) (‘a/b/c’, ‘d.fits’, ‘a/b/c/d.fits’) print get_pathname(‘http://some.where:1234/e/f/g.fits’) (‘e/f’, ‘g.fits’, ‘e/f/g.fits’) print get_pathname(‘h.fits’) (‘’, ‘h.fits’, ‘h.fits’)
common.net.support.test_path(prefix, filename, directories)

Return complete path to a file, if it exists in given directories

The prefix is prefixed to each directory in directories. The directories argument contains a list of directories.

Module contents

Network