common.services.general package

Submodules

common.services.general.HtmlBase module

File HtmlBase.py Author WJ Vriend (wjvriend[AT]astro.rug.nl) Date Feb 2005

class common.services.general.HtmlBase.HtmlBase(args, package_file, parent=None)

Bases: object

This is the base class for a HTML generation class. The functions (interface) defined here are used by the myHttpRequestHandler class

This class should NOT be instantiated, but should be DERIVED from

DEFAULT_PAGE = '<html><body><p>This is the HtmlBase class.</p></body></html>'
SessionLifeTime = None
author_email = ''
author_name = ''
classmethod cleanupSessions()

(override) clean obsolete User Session data

classmethod cleanupUserSession(global_data)

(override) clean specific User Session data data due to time out of SessionLifeTime

cookieFromArgs(cookie_name, cookie_lifetime, arg_names)

make cookie with given name and lifetime from the arg names

cookie_attrs = ('project',)
cookie_lifetime = 365
cookie_name = 'configuration'
default_heading = '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"\n"http://www.w3.org/TR/html4/loose.dtd">\n<html>\n<head>\n <title>%s</title>\n <meta http-equiv="content-type" content="text/html; charset=utf-8">\n <meta http-equiv="Content-Script-Type" content="text/javascript">\n %s\n'
encapsulateBodyHtml(html, title='', extra_head='')

encapsulate the given (body) html with html, head and body tags

encode_default_compress = True
classmethod escapePath(path_var)

escape the possible in a path

getAdditionalHeaders()

(override) return additional http headers

getArg(arg, default='', unquote=False, use_default_for_empty=True)

Get an argument from the list named arg default if argument not present or empty return defualt unquote urllib unquote the value use_default_for_empty

(True) use default for not existing arg and empty values (False) use default only for not existing arg
getCommonPath(service='general')

return the path to the general directory in the common checkout

classmethod getContentLength()

Default the content length is determined from the content, if the content generator writes directly to wfile then this is not possible, override this method to return the content-length in case of using wfile.

classmethod getContentType()

(override) return the http content-type header, can be redefined in derived class

getCookies()

(override) return optional cookie_name, cookie_val, expire, domain

getHtml()

(override) return the generated html, return None to indicate that the generator write to wfile

getHttpServerVar(key, default='')

get the variable of the http server with name ‘key’

getHttpServiceList()

return the service list

classmethod getStatusCode()

(override) return the http status code default value 200 is ok (no errors)

getTemplate(template_filename)

gets the template file object specified by the template_filename

getUserSession(session_id)

set global (persistent) data for the request

classmethod get_url()

(override) return the url for the web service

handler_database = True
hasArg(arg)

test for argument arg

log_enabled = True
log_prefix_address = False
log_prefix_address_awanonymous = True
mangleEmail()

Mangle the email address and put in given message, show message using javascript, msg should contain a %s for the email address.

classmethod package_home()

extract the path name from the globals

setUserSession(data, session_id=None, lock=True)

set global (persistent) data for the request and return the session_id

classmethod url_decode_object(obj_string, compress=True)

Decode the string to a python object Optionally specify to de-compress (zlib) the input string

classmethod url_encode_object(obj, compress=True)

Encode the object for use in url and html forms Optionally specify to compress (zlib) the resulting string

verbose = False
write_wfile(wfile)

(override) write directly to client using the wfile attribute return number of bytes written This method is only called when getHtml() retuns an empty string !

common.services.general.HtmlCluetips module

class common.services.general.HtmlCluetips.HtmlCluetips(cluetips)

Bases: object

class generates html cluetips

Cluetips show a popup window when the user hovers over an (html) element. The cluetip content is defined in a separate <div>, the element on which to show the cluetip is coupled to this div with the rel attribute. Example :

<html> <head> … include jquery.js and jquery-ui.js </head> <script> $(document).ready(function() {

$(‘.local-cluetip’).cluetip({width:400,local:true,sticky:false,clickThrough:true,cluetipClass:’jtip’,cursor:’pointer’});

}); </script> <body> … page …

<p class=”local-cluetip” title=”Object Details” rel=”#cluetip_object_123”>test</p>

… more page …

<div id=”cluetip_local” style=”display:none;”> <div id=”cluetip_object_123”>

<p>The cluetip content shown in a popup</p>

</div> … more cluetips … </div> </body></html>

HTML_CLUETIP = 'class="load-local-info" title="%s|%s" '
HTML_CLUETIP_REL = 'class="load-info" title="%s" rel="#%s"'
classmethod add_cluetip_dict(title, data, cluetip_dict, key_template='localkey_')
getHtml()

generate and return the html

classmethod make_cluetip_html(cluetip_dict, key_template='localkey_')
template = '\n<div id="cluetip_local" style="display:none;">\n${_BEGIN_CLUETIP_DIVS}\n<div id="${CLUETIP_ID}" class="ProcessCluetip">\n${CLUETIP_CONTENT}\n</div>\n${_END_CLUETIP_DIVS}\n</div>\n'

common.services.general.HtmlConfig module

File HtmlConfig.py Author WJ Vriend (wjvriend@astro.rug.nl)

class common.services.general.HtmlConfig.HtmlConfig(args)

Bases: common.services.general.HtmlBase.HtmlBase

This class generates html for setting switches in the http server - verbose : turn on/off verbose logging - SQL_log : turn on/off sql logging - performace : turn on/off performance logging

Config.html.tmpl template file is used

getHtml()

generate html

getToggle(name, default)
initialize()

initialize the toggles, this can not be done __init__

setToggleInTmpl(tmpl, toggle, name, value)

in the template set the name value and button text for the toggle

url_name = 'Config'
class common.services.general.HtmlConfig.HtmlConfigStats(args)

Bases: common.services.general.HtmlBase.HtmlBase

flot_javascript = '\n<script type="text/javascript">\n$(function () {\n%(data_def)s\n\n function doPlot(position) {\n $.plot($("#placeholder"), [\n%(data_plot)s\n ],\n {\n yaxes: [ { // align if we are to the right\n alignTicksWithAxis: position == "right" ? 1 : null,\n position: position,\n } ],\n /*legend: { position: \'nw\',},*/\n grid: {backgroundColor: "rgb(255,255,255)",},\n points: { show: true },\n lines: { show: true },\n xaxis: {min: 0, max: 31},\n });\n }\n doPlot("right");\n});\n</script>\n'
getGraph(stats)

return generated graph

getHtml()

generate the html

url_name = 'ConfigStats'
common.services.general.HtmlConfig.convert(data)

Decode all bytes entries in the given dict. The dict contains user input entries (from project or username), which sometimes contain non ascii characters, which have not been correctly stores in the past, so decode with errors=replace.

common.services.general.HtmlConfig.x_byte(size)

translate value to x KB, MB, GB, TB, etc …

common.services.general.HtmlDBinfo module

File HtmlDBinfo.py Author WJ Vriend (wjvriend@astro.rug.nl)

class common.services.general.HtmlDBinfo.HtmlDBinfo(args)

Bases: common.services.general.HtmlBase.HtmlBase

This class generates the html page for viewing all object definitions in the database.

template file - DBinfo.html.tmpl

getHtml()

generate HTML based on the template file

setColumnDef(tmpl)
setColumnOccurences(tmpl)
setColumnStats(tmpl)
setNameValue(tmpl, sql, block, args)
setTableDef(tmpl)
setTableDetails(tmpl)
setTypeDetails(tmpl)
setTypeRef(tmpl)
setVisibilityColumn(tmpl)
setVisibilityTable(tmpl)
setVisibilityType(tmpl)

common.services.general.HtmlDir module

File HtmlDir.py Author WJ Vriend (wjvriend@astro.rug.nl)

class common.services.general.HtmlDir.HtmlDir(args)

Bases: common.services.general.HtmlBase.HtmlBase

This class generates a html page listing the contents of a directory. template file - Dir.html.tmpl

Only directories relative to the working dir of the server can be requested.

filter_filename(filename)

return boolean to exclude filename this method can be overridden

getHtml()

generate HTML based on the requested dir, using the template file

handler_database = False
list_filename(filename_list)

change the list of filenames this method can be overridden

common.services.general.HtmlError module

File HtmlError.py Author WJ Vriend wjvriend(at)astro(dot)rug(dot)nl

class common.services.general.HtmlError.HtmlError(args)

Bases: common.services.general.HtmlBase.HtmlBase

Class for returning an error in html format

HTML_ERROR = '\n<html>\n<head>\n<title>An unhandled exception</title>\n<link href="%(path_awe_css)s" rel="stylesheet" type="text/css">\n<style type="text/css">\ntable {\n border-spacing: 2px;\n font-family : Verdana;\n}\ntable td {\n padding : 1px 5px 1px 5px;\n font-family : Verdana;\n font-size : 8pt;\n white-space : nowrap;\n}\ntable th {\n background : #bbd;\n padding : 1px 3px 1px 3px;\n font-size : 9pt;\n}\n/* alternating rows in (zebra) table */\ntable tr:nth-child(even) {\n background-color: #fff;\n}\ntable tr:nth-child(odd) {\n background-color: #fee;\n}\n</style>\n</head>\n<body>\n<h1>%(heading)s</h1>\n<p>%(description)s</p>\n<ul>\n<li>Page : %(page)s\n<li>Date : %(date)s\n<li>Exception : %(exception)s\n</ul>\n%(rest)s\n</body></html>'
getHtml()

common.services.general.HtmlForward module

Class for generating a html page that forwards to an url

class common.services.general.HtmlForward.HtmlForward(args, package_file, parent=None)

Bases: common.services.general.HtmlBase.HtmlBase

getHtml()

return the html forward page

html_template = '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">\n<html><head>\n<title>Forwarding ...</title>\n<meta http-equiv="refresh" content="0;url=${forward_url}">\n</head>\n<body>\n<p>You are forwarded to <a href="${forward_url}">${forward_url}</a></p>\n</body></html>'

common.services.general.HtmlGpfs module

Module with html generating classes for GPFS

class common.services.general.HtmlGpfs.Gpfs_basic

Bases: object

html(lines)

format the output lines in html

class common.services.general.HtmlGpfs.Gpfs_mmdf

Bases: object

html(lines)

format the output lines in html

class common.services.general.HtmlGpfs.HtmlGpfs(args)

Bases: common.services.general.HtmlBase.HtmlBase

default_dev = '/dev/gpfs2_new'
getHtml()

generate and return html

common.services.general.HtmlInfo module

File HtmlInfo.py Author WJ Vriend (wjvriend@astro.rug.nl)

class common.services.general.HtmlInfo.HtmlInfo(request, client_address, server)

Bases: http.server.BaseHTTPRequestHandler

Example how to use templates and a http server in python

template file - Info.html.tmpl

do_GET()

common.services.general.HtmlMaintenance module

File HtmlMaintenance.py Author WJ Vriend wjvriend(at)astro(dot)rug(dot)nl

class common.services.general.HtmlMaintenance.HtmlMaintenance(args)

Bases: common.services.general.HtmlBase.HtmlBase

Class for displaying a maintenance message when web server is in maintenance mode. Maintenance mode is enabled when an non empty file called ‘maintenance’ exists in the web service root dir. The contents of this file will displayed.

The contents of the file can contain http status codes, in general : http code TEXT

Supported status codes : 301 Location -> will return “http moved permanently” to the given location

In the above cases the maintenance file must be only one line

getAdditionalHeaders()

return additional http headers for specific http codes

getHtml()

return the maintenance message

getStatusCode()

if specified return custom http status code

maintenance_html = '\n<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"\n "http://www.w3.org/TR/html4/loose.dtd">\n<html>\n<head>\n<title>Web Service in Maintenance</title>\n<link href="%s/awe.css" rel="stylesheet" type="text/css">\n</head>\n<body>\n<h1>Web Service in Maintenance</h1>\n<p>%s</p>\n</body></html>'

common.services.general.HtmlProjects module

File HtmlProjects.py Author WJ Vriend (wjvriend[AT]astro.rug.nl)

Class gives an overview of all projects in the database

class common.services.general.HtmlProjects.HtmlProjects(args, parent=None)

Bases: common.services.general.HtmlBase.HtmlBase

This class generates a html page listing the projects in the database

template file - Projects.html.tmpl

This can be a stand-alone page (including html and body tags) or included in another page.

To include this in another page set ‘full_html’ to ‘0’ in the args dict, and provide a css class for the table, ‘table_extra’ in the args dict.

default_order_by = 'ID'
getHtml()

generate HTML based on the requested dir, using the template file

getHtmlProjectTable(projects, table_id='project_table', table_extra='')

return the html table for the given projects

getProjectUsers()

get the dictionary with the project - users

getTableGroups()

return the grouping of tables

getTopPage(table_groups_html)

return the top of the html page

get_additional_td(project_id)

(override) to return additional table row elements

get_additional_th()

(override) to return additional table headers

heading = '\n<link href="%(common_path)s/services/css/awe.css" rel="stylesheet" type="text/css">\n\n<!-- See webpage of jquery http://jquery.com/ -->\n<script type="text/javascript" src="%(common_path)s/services/javascript/jquery.js"></script>\n<script type="text/javascript" src="${common_path}/services/javascript/jquery-ui.js"></script>\n<link type="text/css" rel="stylesheet" href="${common_path}/services/css/jquery-ui.css">\n\n<style type="text/css">\n/* definition for project table */\ntable {\n border-spacing: 2px;\n/* font-family : Verdana;*/\n}\ntable td {\n font-family : Verdana;\n font-size : 8pt;\n}\ntable th {\n background : #bbd;\n font-family : Verdana;\n font-size : 9pt;\n}\n</style>\n'
makeUserTable(users)

make html table with all users of the project

managers_in_tooltip = False
page_top = '\n<h2>Projects of %(db_name)s</h2>\n\n<p>\nTotal number of projects %(nr_projects)s<br>\nTotal number of users %(nr_users)s\n</p>\n%(project_click)s\n%(table_groups_html)s\n'
classmethod project_table_id(table_group='')

return the id of the project table if table_group is given, return the id of project group table

classmethod remove_from_url_query(url, remove_arg='project')

remove arg (default project) from the url query part

skip_project(project_id, project_name, description, instrument, privileges)

(override) return True to skip this project

common.services.general.HtmlTemplate module

File HtmlTemplate.py Author WJ Vriend (wjvriend@astro.rug.nl) Date Dec 2004

class common.services.general.HtmlTemplate.HtmlTemplate(TemplateFile, template_string='')

Bases: object

Html Template class reads a template file and substitutes variables with values where variables can be placed inside repeating blocks

—– ===== Example 1 (Creating a template object) ===== —–

# from a file tmpl = HtmlTemplate(a_template_filename) # from a string tmpl = HtmlTemplate(None, template_string=”<html>…${VAR}…</html>”)

—– ===== Example 1 (filling in variables) ===== —–

[HTML Template]

<html><body>
${VAR}

</body></html>

[PYTHON]

tmpl = HtmlTemplate([HTML Template]) tmpl.setVariable( ‘VAR’, ‘Hello World’) # will replace ${VAR} with TEST tmpl.getHtml()

[HTML output]

<html><body>
Hello World

</body></html>

—– ===== Example 2 (doing blocks) ===== —–

[HTML Template]

<html><body> ${_BEGIN_BLOCK}

${VAR}

${_END_BLOCK} </body></html>

[PYTHON]

tmpl = HtmlTemplate([HTML Template]) a_block = tmpl.beginBlock(‘BLOCK’) a_block.setBlockVariable(‘VAR’, ‘Hello World 1’) a_block.updateBlock() a_block.setBlockVariable(‘VAR’, ‘Hello World 2’) a_block.updateBlock() a_block.setBlockVariable(‘VAR’, ‘Hello World 3’) a_block.updateBlock() tmpl.insertRepeatingBlock(a_block) tmpl.getHtml()

[HTML output]

<html><body>
Hello World 1 Hello World 2 Hello World 3

</body></html>

—– ===== Example 2 (doing blocks in blocks) ===== —–

[HTML Template]

<html><body> ${_BEGIN_BLOCK_OUTER}

${VAR_OUTER} ${_BEGIN_BLOCK_INNER}

${VAR_INNER}

${_END_BLOCK_INNER}

${_END_BLOCK_OUTER} </body></html>

[PYTHON]

tmpl = HtmlTemplate([HTML Template]) a_outer_block = tmpl.beginBlock(‘BLOCK_OUTER’) a_outer_block.setBlockVariable(‘VAR_OUTER’, ‘Hello Outer World 1’) a_inner_block = a_outer_block.beginBlock(‘BLOCK_INNER’) a_inner_block.setBlockVariable(‘VAR_INNER’, ‘Hello Inner World 1’) a_inner_block.updateBlock() a_inner_block.setBlockVariable(‘VAR_INNER’, ‘Hello Inner World 2’) a_inner_block.updateBlock() a_outer_block.insertRepeatingBlock(a_block) a_outer_block.updateBlock() a_outer_block.setBlockVariable(‘VAR_OUTER’, ‘Hello Outer World 2’) a_inner_block = a_outer_block.beginBlock(‘BLOCK_INNER’) a_inner_block.setBlockVariable(‘VAR_INNER’, ‘Hello Inner World 3’) a_inner_block.updateBlock() a_outer_block.insertRepeatingBlock(a_block) a_outer_block.updateBlock() tmpl.insertRepeatingBlock(a_block) tmpl.getHtml()

[HTML output]

<html><body>
Hello Outer World 1
Hello Inner World 1 Hello Inner World 2
Hello Outer World 2
Hello Inner World 3

</body></html>

beginBlock(BlockName)

initialize a block

endBlock()

copy the filled blocks to the html file

generate()

generate the content from the template

getHtml()

return the content(html)

insertBlock(block_name, block_content)

insert block_content into the template where block_name

insertRepeatingBlock(repeating_block)

insert repeating block in template

openTemplateFile(TemplateFilename)

open a template file and read content

replaceInContent(varName, varValue)

replaceInContent will replace the given string in the template content with the given value

saveHTML(filename)

save the the html to a file

setBlockValues(blockName, blockDicts)

setBlockValues will replace the variables in the template blockName with the given list of name,value pairs in blockDicts

setBlockVariable(varName, varValue)

set a variable within this block

setVariable(varName, varValue, overwrite=True)

setVariable will replace the variable in the template file with the given value

setVariableDict(varValuesDict, overwrite=True)

setVariableDict will replace the variables in the template file with the given values

setVariableList(varNames, varValues, overwrite=True)

setVariableList will replace the variables in the template file with the given values

updateBlock()

update the block template

warn_on_not_used_vars = True
class common.services.general.HtmlTemplate.RepeatingBlock(name, template)

Bases: object

beginBlock(BlockName)

initialize a repeating block

insertBlock(block_name, block_content)

insert block_content into the template where block_name

insertRepeatingBlock(repeating_block)

insert repeating block in template

setBlockVariable(varName, varValue)

set a variable within this block

updateBlock()

copy the current filled block to the content

exception common.services.general.HtmlTemplate.TemplateError

Bases: Exception

common.services.general.HtmlTemplate.beginblock(name)

return begin block name as used in template

common.services.general.HtmlTemplate.endblock(name)

return end block name as used in template

common.services.general.HtmlTemplate.get_begin_end(content, BlockName)

get the begin and end position of block

common.services.general.HtmlTemplate.varname(var_name)

return var name as used in template

common.services.general.HtmlTest module

class common.services.general.HtmlTest.HtmlTest(args)

Bases: common.services.general.HtmlBase.HtmlBase

getHtml()

generate the html

common.services.general.HtmlUtil module

Collection of html (generation) utilities

class common.services.general.HtmlUtil.HtmlUtil

Bases: object

EOL = '\n'
classmethod Escape(text)

escape html characters

classmethod UnEscape(text)

un-escape html characters

classmethod escapeOverlib(text)

overlib expects single quotes as ‘ and double quote as &quot;

classmethod generateOption(value, display, selected, disabled, extra='')

make a html <option value=..>..</option> and selecting or disabling the option

classmethod generatePrivateProjectSelect(sel_project, max_len=25, valid_instruments=[])

generate a html select box with all private projects

classmethod generateProjectSelect(privileges, sel_project, max_len=25, valid_instruments=[])

Generate html select box with projects with specified privileges

classmethod generatePublicProjectSelect(sel_project, max_len=25, valid_instruments=[])

generate a html select box with all public projects

classmethod generateSelect(name='', display=[], values=[], selected='', disabled=[], extras=[], select_name_values={})

make a html <select>…</select> from the display and values lists selecting the selected

classmethod generateSelectOptions(display=[], values=[], selected='', disabled=[], extras=[])

make a html collection of <option>…</option> from the display and values lists selecting the selected and disabling the disabled

classmethod make_radio_buttons(name, values, selected, displays=None)

Return html radio button definitions with name, one for each value . The radio button with value selected is selected. If displays is given then that will be shown as text, otherwise the values will be shown.

classmethod return_radio_checked(check, choice)

return radio choice

common.services.general.HttpBots module

Module with HttpBots class for bot detection

WJ Vriend

class common.services.general.HttpBots.HttpBots

Bases: object

Class for testing if (http) client is a bot

isBot(ip, headers)

Is the given name a bot name ?

testValue(value, re_values)

Test all the re_names against the test name

useragents_file = 'HttpBots.cfg'

common.services.general.HttpCleanup module

Module with (example) Cleanup class for cleaning log files in the debug dir

WJ Vriend

class common.services.general.HttpCleanup.Cleanup(server)

Bases: object

(example) Class for cleaning up log files

Other Cleanup implementations must at least :
the __init__ method must accept the server object as argument the do_work method must be defined
cleanup()

remove old files

do_work()

method which is called every X seconds

get_log_pathname()

return the full log pathname

lock_name = 'Cleanup'
log_basename = 'CleanupThread.log'
long_log_age = datetime.timedelta(100)
short_log_age = datetime.timedelta(1)

common.services.general.HttpConfig module

Class for handling the http(s) configuration

WJ Vriend

class common.services.general.HttpConfig.ConfigParserStripQuotes(defaults=None, dict_type=<class 'collections.OrderedDict'>, allow_no_value=False, *, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section='DEFAULT', interpolation=<object object>, converters=<object object>)

Bases: configparser.RawConfigParser

ConfigParser that strips quotes from values

get(section, option, *args, **kwargs)
items(section, *args, **kwargs)
stripQuotes(val)

strip single or double from value

class common.services.general.HttpConfig.HttpConfig(options)

Bases: object

check_config(config_dict)

check the config dict

check_env()

check the Environment

configuration_filename()

determine the configuration filename

configuration_parser()

return the configuration parser

getOption(option, default)

retrieve value of option, if not present use default

classmethod get_startup_args()

Return the startup arguments for the HTTP(S) server

classmethod import_class(class_specification)

import the class from the class specification, and return the python class

init_access_list(config_parser)

initialize the access_list from the config file

classmethod init_cleanup_thread(server)

init the cleanup (worker) thread specified in the config file (optional)

init_config(config_parser)

init the config struct from - the configuration file - the command line

init_host_mapping(config_parser)

initialize the host mapping dictionary

init_service_list(config_parser)

initialize the service list, dictionary coupling the html generation class to a http request

init_sessions(http_config)

initialize the login Sessions

make_server()

do all the configuration

exception common.services.general.HttpConfig.HttpConfigError

Bases: Exception

common.services.general.HttpDatabase module

Database for the http server and services

class common.services.general.HttpDatabase.HttpDatabase(filename)

Bases: object

init_tables(conn)

initialize database tables

insert_query(query, args, duration, session=-1)

insert query row in database

tables = {'queries': (('created', 'timestamp'), ('query', 'text'), ('args', 'text'), ('username', 'text'), ('project', 'text'), ('privilegs', 'int'), ('http_session', 'int'), ('duration', 'real'))}

common.services.general.HttpFileRequest module

module with HttpFileRequest class

WJ Vriend

class common.services.general.HttpFileRequest.HttpFileRequest(args)

Bases: common.services.general.HtmlBase.HtmlBase

Class for handling http file requests

content_types = {'.css': 'text/css', '.fits': 'application/fits', '.gif': 'image/gif', '.gz': 'application/octet-stream', '.htm': 'text/html', '.html': 'text/html', '.ico': 'image/x-icon', '.jpeg': 'image/jpeg', '.jpg': 'image/jpeg', '.js': 'application/x-javascript', '.log': 'text/plain', '.png': 'image/png', '.txt': 'text/plain', '.xml': 'text/xml'}
correctRootedPath(path)

The http server serves files from either the root dir (/) or a virtual root (/home/user/static/) If virtual_root is specified in the dir_list section then we should only serve files from this root. If virtual_root is not specifed in the dir_list section then the access_list section should be used to limit the files being served. (though this mode should be considered obsolete) Path can be given in two manners; old and new style

new : [/]common/services/css/awe.css old : /full/path/to/repository/common/services/css/awe.css
default_content_type = 'application/octet-stream'
doesExist()

does the request exist and what is the absolute path

existAndAllowed()

does the request exists and is allowed to serve ?

expire_delta = datetime.timedelta(1)
extraAllowedCheck()

(derive) Derived classes can implement this method to do extra allowed checks

getAdditionalHeaders()

return headers for this file request

getContentType()

return the http content-type header

getHtml()

file is written directly to wfile, see write_wfile method

getStatusCode()

return the http status code

handler_database = False
http_date_format = '%a, %d %h %Y %T GMT'
isAllowed()

is the request allowed to serve ?

log_enabled = False
readsize = 10000000
write_dump_log()

write message to the dump log

write_wfile(wfile)

file is requested, read and return the content determine the filetype and return metadata of the file

common.services.general.HttpSessionPool module

Module with the HttpSessionPool class

class common.services.general.HttpSessionPool.HttpSessionPool(lock)

Bases: dict

Class for interacting with persistent data (serverside) for user sessions

The class itself is a dictionary storing per HtmlBase class a dictionary with session_ids pointing to a dictionary with session data :

self[html_class][session_id][…]

The … can be used by the session to store data

Reserver keywords in this dict are :
date_started - when the session was started date_modified - last change in session
cleanup()

remove all the SessionPool objects older then the SessionLifeTime, the associated resources are removed by the objects cleanupUserSession method

destructor()

cleanup

getUserSession(cls, session_id)

get the global data for the class (cls) and user (ip,port)

initSession(ip, port, data)

initialize a new session and return new session id

lock_name = 'HttpSessionPool'
makeSessionId(ip, port, now)

generate session_id

setUserSession(cls, session_id, data)

store the global_data

common.services.general.HttpStatistics module

Module containing HttpStatistics class

class for handling statistics of http server

class common.services.general.HttpStatistics.HttpPipeListener(server, debug=False)

Bases: threading.Thread

Thread class listens on named pipe for statistic info, when the HTTP service is run in forked then the child processes use this to send statistics to the parent process.

run()

run the thread

stop_command = 'STOP'
class common.services.general.HttpStatistics.HttpStatistics(config, lock)

Bases: dict

addStat(stat, value, username, project)

add the value of the stat to the internal dict

appendStat(stat, value, username, project)

Append the value to the internal dict

corrupt_stats_mail = 'The statistics file "%s" is corrupt. Restore backup or statistics history is lost.'
day_date_format = '%Y-%m-%d'
init_from_old_style(old_style_stats)

init from old style stats

initialize()

clean initialize

lock_name = 'HttpStatistics'
maxlen = 10
saveStatistics()

save the statistics to file

class common.services.general.HttpStatistics.HttpStatisticsSimple(server_stat_file, lock)

Bases: object

Class for collecting (simple) statistics, only count nr of GET, POST, files, etc …

addStat(stat, value=1)

Add the value to that statistic named stat, also add to the forever value

appendStat(stat, value)

Append the value to the statistic named stat

count_items = ['do_CONNECT', 'do_DELETE', 'do_GET', 'do_HEAD', 'do_OPTIONS', 'do_PATCH', 'do_POST', 'do_PUT', 'do_TRACE', 'request_invalid', 'request_not_existing', 'number_of_bytes', 'number_of_files', 'request_count']
getStat(stat, forever=False)

return the value for the given stat

getStats()
ignore_items = ['forever_do_delete', 'forever_do_get', 'forever_do_head', 'forever_do_post', 'forever_do_put']
init_server_stat(stats, forever_items, stat_filename)

initialize the server stat file

init_stats(server_stat_file)

initialize the statistics

lock_name = 'HttpStatisticsSimple'
maxlen = 10
newStat(stat, value)

create new stat with value

removeStat(stat)

remove the given stat

common.services.general.HttpThreadLock module

Http thread lock class

class common.services.general.HttpThreadLock.HttpDummyLock

Bases: object

Dummy Locking class for the Http Server, this just raises an error when used

acquire(name)

acquire the lock

release(name)

release the lock

class common.services.general.HttpThreadLock.HttpThreadLock(verbose=False)

Bases: object

Thread Locking class for the Http Server

acquire(name)

acquire the lock

release(name)

release the lock

common.services.general.HttpTimer module

Module with the HttpTimer class

class common.services.general.HttpTimer.HttpTimer(server, worktime=None)

Bases: object

Class for doing timed actions in http server

  • does cleanup of user sessions in session_pool
  • updates the statistics file
  • mail the error log if errors occured in the last 24 hours
  • cleanup of log files
WorkTime = 300
destructor()

cleanup

do_work(last=False)

method which is called every self.WorkTime seconds if argument last is True then stop after this call

log_filename = 'http_timer_error.log'
classmethod mail(from_address, to_address, subject, content, mail_server='smtp.astro-wise.org')

send mail

mail_error_file()

mail the error log file in case of errors (once a day)

start_timer()

start the timer

write_stats()

write the forever stats to file

exception common.services.general.HttpTimer.HttpTimerError

Bases: Exception

common.services.general.SqlUtil module

Collection of sql methods for web services

class common.services.general.SqlUtil.SqlUtil

Bases: object

SqlUtil class with sql helper methods for web services

classmethod pageSql(sql, rowmin, rowmax)

encapsulate query to enable paging (next page) see http://www.oracle.com/technology/oramag/oracle/06-sep/o56asktom.html

common.services.general.TextMultipleUrls module

TextMultipleUrls.py

return an input file for wget to download multiple files/urls

WJ Vriend

class common.services.general.TextMultipleUrls.TextMultipleUrls(args)

Bases: common.services.general.HtmlBase.HtmlBase

Class for generating text file with multiple urls

input :

download_filename the filename of the script (default urls.txt)

download_urls list of pickled urls

< or >
download_files list of pickled filenames download_server the hostname of the server to download from
getAdditionalHeaders()

return additional header item(s)

getContentType()

return the content-type header

getHtml()

generate html

common.services.general.TextUtil module

Methods for text handling and formatting

WJ Vriend

common.services.general.TextUtil.x_byte(size)

translate value to human readable form : x KB, MB, GB, TB, etc …

common.services.general.ThreadedHTTPServer module

ThreadingHTTPServer is based on ThreadingTCPServer and ForkingHTTPServer is based on ForkingTCPServer.

The Threading class starts a new thread for each request (although multpiple request from one client are still done sequential ?), the forking class forks for each request.

Threading is very light in resources, forking heavy. But in case of threading the memory is shared between threads, which can result in seemingly random, very hard to find bugs.

Testing showed that the processing time for the threading and processing model (forking) is: - Threading 0.0002 to 0.0003 seconds - Processing 0.01 to 0.1 seconds

A simple html page was used as test object, the server and the client both run on the same machine.

WJ Vriend (wjvriend@astro.rug.nl)

TODO - min / max amount of threads / processes - queuing of requests in existing threads / processes - get max_children and ALLOW_REUSE_ADDRESS from configuration

class common.services.general.ThreadedHTTPServer.ForkingHTTPServer(server_address, RequestHandlerClass, bind_and_activate=True)

Bases: socketserver.ForkingTCPServer

allow_reuse_address = 1
collect_children(finished_only=False)

This overwrties the method collect_children in SocketServer.py

If finished_only is set then only the finished (and defunct) processes are handled otherwise the original collect_children is called.

max_children = 100
class common.services.general.ThreadedHTTPServer.ThreadingHTTPServer(server_address, RequestHandlerClass, bind_and_activate=True)

Bases: socketserver.ThreadingTCPServer

allow_reuse_address = 1
collect_children(finished_only=False)

Dummy method, making the interface similar as ForkingHTTPServer

daemon_threads = True
common.services.general.ThreadedHTTPServer.discard_finished_children(active_children)

Iterate all children, and discard the ones that are finished

common.services.general.myColors module

common.services.general.myColors.darkenColor(color, factor)

darken color by factor

common.services.general.myColors.joinHexRGBColor(r, g, b)

join the rgb components t oa color

common.services.general.myColors.splitHexRBGColor(color)

split the given color in the rgb components

common.services.general.myDB module

File myDB.py Author WJ Vriend (wjvriend@astro.rug.nl) Date Dec 2004

common.services.general.myDB.get_oid(table_alias)

return object_id clause for use in sql

common.services.general.myDB.sql_table(table_name)

construct table name for sql

common.services.general.myHttpRequestHandler module

File myHttpRequestHandler.py Author WJ Vriend wjvriend(at)astro(dot)rug(dot)nl Date Dec 2004

Python code for handling HTTP requests in a HTTP(S) Astro-WISE server

Configuration :

See HttpServer.cfg for the configuration options, this configuration file (global) should exist in the same directory as the current file. With the -c option a custom configuration file can be specified.

See class myHttpRequestHandler for more information

class common.services.general.myHttpRequestHandler.myHttpRequestHandler(request, client_address, server)

Bases: http.server.BaseHTTPRequestHandler

Description:

Handles post, get and head requests as a Http server.

myHttpRequestHandler class is created by BaseHTTPServer (http_server) everytime a http request is made.

For each request 1. a class is instantiated with the given parameters 2a. the class method getContentType is called for the content-type 2b. the class method getAdditionalHeaders is called for optional headers 2c. the class method getCookies is called for optional cookies 3. the class method getHtml() is called to generate the html page

Static file requests are handled by the http server itself.

The http server has three toggles: -verbose For (very) verbose output (TODO; couple this to the log object) -performance For every request the processing time is logged -sql Logs every sql statement (TODO; couple this to the DB layer)

The following parameters will be used by the server for fillign the Profile : - project The project to be set in the Profile - privileges The privileges to be set in the Profile - database_name The database name (or connection string) to use - username The username for making the database connection

The following parameters are always set by the http server in the args dictionary: * _maintenance_lines * _request * _request_method * _http_error

The html generation classes can specify the following attributes : - handler_database create a database connection for this request ?

For storing data between sessions the session_pool can be used.

The server keeps track of statistical information since last start and forever.

The coupling of request and class is hard-coded, see the service_list for the coupling. The service list is defined in the configuration file. If enabled then a status page of the web server can be requested using the url “http://server:port/Config”.

myHttpServer startup syntax:

>awe $AWEPIPE/common/services/general/myHttpServer

-c configration file
-h hostname
-p port is the port number the server listens on
-d domain is the domain the server is part of (example astro-wise.org) this is used for the cookies
-s default_service is one of the above stated requests. This will be be served when nothing speicific (or /) is requested.
-t test mode, dump all logging on the screen
-v (set) version

If an non empty file called ‘maintenance’ exists in the root of the web server then the contents of this file will be showed for every request.

Process(method)

main method for handling a request

addStat(stat, value=1)

Add the value to that statistic named stat, also add to the forever value

appendStat(stat, value)

Append the value to the statistic named stat

classmethod cleanLogLine(line)

Clean the log line, see cleanLogLineArg()

classmethod cleanLogLineArg(arg)

Clean the log line argument, remove password entries

cleanup()

stop close http fd’s, DB, stop logging, remove profile

classmethod cls_getCookieName(cookie_name, server_name, old_style=False)

Return the cookie name to use, prepended with the server name Server name and cookie were joined with “_”, from 9 Feb “#” is used (so server name can contain “_”)

connectDb(profile)

try to connect to the database, using the given profile

connectDbAttempt(profile)

try to connect, log if failure

cookie_aw_session_expires = 1
cookie_max_size = 4000
cookie_sso_expires = 24
do_CONNECT()

handle CONNECT request, is same as do_GET ? (not tested)

do_DELETE()

handle DELETE request, is same as do_GET ? (not tested)

do_GET(method='GET')

handle a GET request

do_HEAD()

handle a HEAD request, same as GET but only returning headers

do_OPTIONS()

handle OPTIONS request, is same as do_GET ? (not tested)

do_PATCH()

handle PATCH request, is same as do_GET ? (not tested)

do_POST()

handle a POST request

do_PUT()

handle PUT request, store PUT information in _put_file as file object

do_TRACE()

handle TRACE request, is same as do_GET ? (not tested)

finish()

Called by BaseRequestHandler when request is finished

classmethod form_to_dict(form_data)

put form data in dict, form elements with name can have multiple values

formatError(error)

Format the error message, this is called in case of unhandled expection, In case of custom error call this method to format the error message

getActiveCount()

return the number of active threads or processes

getClientIp()

determine and return the IP address of the client the IP can be obscured by a proxy

getCookieName(name, old_style=False)

See class method version; cls_getCookieName

getCookieValue(name)

If cookie with name exists return value, otherwise None Cookie names are stored with the server name prepended

getCookiesForService(service_name)

return the cookies meant for given service

getDataForLog()

Return the data dictionary prepared for log

getHtmlGenerator()

Return the object handling this request

getHtmlGeneratorClass()

Return the class handling this request

getStat(stat, forever=False)

return the value for the given stat

getUrl()

Return the full url of this request

getUrlParts()

Return the full url of this; protocol, server, path

getUserSession(session_id, session_class=None)

return the global (persistent) data for the request

get_log_date_adress()

start of every log line

handleCookies()

fill self.cookies dict with all cookies meant for this service

handleInput(args)

handle the input arguments dictionary, decode the binary values to string

handle_exception(exception)

handle exceptions, if user cancels request, log it and continue, else format exception in html

initRequest(method)

initialize for the request, there can be multiple requests handled by one request handler instance

initRequestPath()

initialize the request path; unquote and possibly add path

init_profile()

set the profile; variables username, password, project

logError(error)

write error message to the error log file

log_message(format, *args)

called from BaseHTTPRequestHandler, logging method for main thread, some (boring) log is writtten to dump file the rest is printed, use Message() to log thread specifics

makeCookie(cookie)

make a cookie string from either a list/tuple or dict (preferred)

newStat(stat, value)

create new stat with value

removeStat(stat)

remove the given stat

sendHeaders(content)

send the http headers

setUserSession(global_data, session_id=None, lock=True, session_class=None)

set the global (persistent) data for the request

start_log(profile)

start logging for this request

status_codes_ok = (200, 201, 301, 302, 303, 307, 308)
tryProcess(request, method)

try processing the request

useCookies()

place the name value pairs from the cookies in the data dictionary and extract the username and password from the possible login cookies

writePipe(action, stat, value, username, project)

write the stat and value on the named pipeline

exception common.services.general.myHttpRequestHandler.myHttpRequestHandlerError

Bases: Exception

common.services.general.myHttpServer module

File myHttpServer.py Author WJ Vriend wjvriend(at)astro(dot)rug(dot)nl Date Dec 2004

Python code for running an HTTP(s) server in the Astro-Wise environment

Configuration :

See HttpServer.cfg for the configuration options, this configuration file (global) should exist in the same directory as the current file. With the -c option a custom configuration file can be specified.

Enabling HTTPS :

* Certificates *

When using ssl a private key and certificate are required, these are the steps to create them :

openssl genrsa -des3 -out server.key 1024 openssl req -new -key server.key -out server.csr openssl rsa -in server.key -out server.key openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

server.crt is the certificate, server.key is the private key see $AWEPIPE/common/services/genereal/HttpServer.cfg for an example of the configuration file under the [ssl] section the key and certificate must be specified

* Keys *

The login service uses the following keys (in Env):

encryption_key; to encrpyt and decrypt the username/passwords login_server_key_init; to allow other services to initiate a Session login_server_key_request; to allow other services to request a Session

Example how to generate random base64 encoded keys # python code import uuid u = uuid.uuid4() u.bytes.encode(“base64”)

common.services.general.myHttpServer.main(arguments)

main method

common.services.general.my_parse_multipart module

Original from cgi.parse_multipart

Added report of upload progress back to handler

To make use of the upload progress provide the following arguments in the POST or GET session:
‘_upload_classname’ and ‘_upload_id’
Then in an UserSession with id ‘_upload_id’ the following information is stored :
‘content-length’ : the total content length (from the header) ‘uploaded’ : the number of bytes uploaded

At the end the ‘uploaded’ will be set to ‘content-length’ (otherwise the ‘uploaded’ will never be equal to ‘content-length’,

because we can not determine the header length)
common.services.general.my_parse_multipart.progress_parse_multipart(fp, pdict, handler, content_len, header_len=0)

Module contents