Submodules module

File Author WJ Vriend (wjvriend[AT] Date Feb 2005

class, package_file, parent=None)

Bases: object

This is the base class for a HTML generation class. The functions (interface) defined here are used by the myHttpRequestHandler class

This class should NOT be instantiated, but should be DERIVED from

DEFAULT_PAGE = '<html><body><p>This is the HtmlBase class.</p></body></html>'
SessionLifeTime = None
author_email = ''
author_name = ''
classmethod cleanupSessions()

(override) clean obsolete User Session data

classmethod cleanupUserSession(global_data)

(override) clean specific User Session data data due to time out of SessionLifeTime

cookieFromArgs(cookie_name, cookie_lifetime, arg_names)

make cookie with given name and lifetime from the arg names

cookie_attrs = ('project',)
cookie_lifetime = 365
cookie_name = 'configuration'
default_heading = '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"\n"">\n<html>\n<head>\n <title>%s</title>\n <meta http-equiv="content-type" content="text/html; charset=utf-8">\n <meta http-equiv="Content-Script-Type" content="text/javascript">\n %s\n'
encapsulateBodyHtml(html, title='', extra_head='')

encapsulate the given (body) html with html, head and body tags

encode_default_compress = True
classmethod escapePath(path_var)

escape the possible in a path


(override) return additional http headers

getArg(arg, default='', unquote=False, use_default_for_empty=True)

Get an argument from the list named arg default if argument not present or empty return defualt unquote urllib unquote the value use_default_for_empty

(True) use default for not existing arg and empty values (False) use default only for not existing arg

return the path to the general directory in the common checkout

classmethod getContentLength()

Default the content length is determined from the content, if the content generator writes directly to wfile then this is not possible, override this method to return the content-length in case of using wfile.

classmethod getContentType()

(override) return the http content-type header, can be redefined in derived class


(override) return optional cookie_name, cookie_val, expire, domain


(override) return the generated html, return None to indicate that the generator write to wfile

getHttpServerVar(key, default='')

get the variable of the http server with name ‘key’


return the service list

classmethod getStatusCode()

(override) return the http status code default value 200 is ok (no errors)


gets the template file object specified by the template_filename


set global (persistent) data for the request

classmethod get_url()

(override) return the url for the web service

handler_database = True

test for argument arg

log_enabled = True
log_prefix_address = False
log_prefix_address_awanonymous = True

Mangle the email address and put in given message, show message using javascript, msg should contain a %s for the email address.

classmethod package_home()

extract the path name from the globals

setUserSession(data, session_id=None, lock=True)

set global (persistent) data for the request and return the session_id

classmethod url_decode_object(obj_string, compress=True)

Decode the string to a python object Optionally specify to de-compress (zlib) the input string

classmethod url_encode_object(obj, compress=True)

Encode the object for use in url and html forms Optionally specify to compress (zlib) the resulting string

verbose = False

(override) write directly to client using the wfile attribute return number of bytes written This method is only called when getHtml() retuns an empty string ! module


Bases: object

class generates html cluetips

Cluetips show a popup window when the user hovers over an (html) element. The cluetip content is defined in a separate <div>, the element on which to show the cluetip is coupled to this div with the rel attribute. Example :

<html> <head> … include jquery.js and jquery-ui.js </head> <script> $(document).ready(function() {


}); </script> <body> … page …

<p class=”local-cluetip” title=”Object Details” rel=”#cluetip_object_123”>test</p>

… more page …

<div id=”cluetip_local” style=”display:none;”> <div id=”cluetip_object_123”>

<p>The cluetip content shown in a popup</p>

</div> … more cluetips … </div> </body></html>

HTML_CLUETIP = 'class="load-local-info" title="%s|%s" '
HTML_CLUETIP_REL = 'class="load-info" title="%s" rel="#%s"'
classmethod add_cluetip_dict(title, data, cluetip_dict, key_template='localkey_')

generate and return the html

classmethod make_cluetip_html(cluetip_dict, key_template='localkey_')
template = '\n<div id="cluetip_local" style="display:none;">\n${_BEGIN_CLUETIP_DIVS}\n<div id="${CLUETIP_ID}" class="ProcessCluetip">\n${CLUETIP_CONTENT}\n</div>\n${_END_CLUETIP_DIVS}\n</div>\n' module

File Author WJ Vriend (



This class generates html for setting switches in the http server - verbose : turn on/off verbose logging - SQL_log : turn on/off sql logging - performace : turn on/off performance logging

Config.html.tmpl template file is used


generate html

getToggle(name, default)

initialize the toggles, this can not be done __init__

setToggleInTmpl(tmpl, toggle, name, value)

in the template set the name value and button text for the toggle

url_name = 'Config'


flot_javascript = '\n<script type="text/javascript">\n$(function () {\n%(data_def)s\n\n function doPlot(position) {\n $.plot($("#placeholder"), [\n%(data_plot)s\n ],\n {\n yaxes: [ { // align if we are to the right\n alignTicksWithAxis: position == "right" ? 1 : null,\n position: position,\n } ],\n /*legend: { position: \'nw\',},*/\n grid: {backgroundColor: "rgb(255,255,255)",},\n points: { show: true },\n lines: { show: true },\n xaxis: {min: 0, max: 31},\n });\n }\n doPlot("right");\n});\n</script>\n'

return generated graph


generate the html

url_name = 'ConfigStats'

Decode all bytes entries in the given dict. The dict contains user input entries (from project or username), which sometimes contain non ascii characters, which have not been correctly stores in the past, so decode with errors=replace.

translate value to x KB, MB, GB, TB, etc … module

File Author WJ Vriend (



This class generates the html page for viewing all object definitions in the database.

template file - DBinfo.html.tmpl


generate HTML based on the template file

setNameValue(tmpl, sql, block, args)
setVisibilityType(tmpl) module

File Author WJ Vriend (



This class generates a html page listing the contents of a directory. template file - Dir.html.tmpl

Only directories relative to the working dir of the server can be requested.


return boolean to exclude filename this method can be overridden


generate HTML based on the requested dir, using the template file

handler_database = False

change the list of filenames this method can be overridden module

File Author WJ Vriend wjvriend(at)astro(dot)rug(dot)nl



Class for returning an error in html format

HTML_ERROR = '\n<html>\n<head>\n<title>An unhandled exception</title>\n<link href="%(path_awe_css)s" rel="stylesheet" type="text/css">\n<style type="text/css">\ntable {\n border-spacing: 2px;\n font-family : Verdana;\n}\ntable td {\n padding : 1px 5px 1px 5px;\n font-family : Verdana;\n font-size : 8pt;\n white-space : nowrap;\n}\ntable th {\n background : #bbd;\n padding : 1px 3px 1px 3px;\n font-size : 9pt;\n}\n/* alternating rows in (zebra) table */\ntable tr:nth-child(even) {\n background-color: #fff;\n}\ntable tr:nth-child(odd) {\n background-color: #fee;\n}\n</style>\n</head>\n<body>\n<h1>%(heading)s</h1>\n<p>%(description)s</p>\n<ul>\n<li>Page : %(page)s\n<li>Date : %(date)s\n<li>Exception : %(exception)s\n</ul>\n%(rest)s\n</body></html>'
getHtml() module

Class for generating a html page that forwards to an url

class, package_file, parent=None)



return the html forward page

html_template = '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">\n<html><head>\n<title>Forwarding ...</title>\n<meta http-equiv="refresh" content="0;url=${forward_url}">\n</head>\n<body>\n<p>You are forwarded to <a href="${forward_url}">${forward_url}</a></p>\n</body></html>' module

Module with html generating classes for GPFS


Bases: object


format the output lines in html


Bases: object


format the output lines in html



default_dev = '/dev/gpfs2_new'

generate and return html module

File Author WJ Vriend (

class, client_address, server)

Bases: http.server.BaseHTTPRequestHandler

Example how to use templates and a http server in python

template file - Info.html.tmpl

do_GET() module

File Author WJ Vriend wjvriend(at)astro(dot)rug(dot)nl



Class for displaying a maintenance message when web server is in maintenance mode. Maintenance mode is enabled when an non empty file called ‘maintenance’ exists in the web service root dir. The contents of this file will displayed.

The contents of the file can contain http status codes, in general : http code TEXT

Supported status codes : 301 Location -> will return “http moved permanently” to the given location

In the above cases the maintenance file must be only one line


return additional http headers for specific http codes


return the maintenance message


if specified return custom http status code

maintenance_html = '\n<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"\n "">\n<html>\n<head>\n<title>Web Service in Maintenance</title>\n<link href="%s/awe.css" rel="stylesheet" type="text/css">\n</head>\n<body>\n<h1>Web Service in Maintenance</h1>\n<p>%s</p>\n</body></html>' module

File Author WJ Vriend (wjvriend[AT]

Class gives an overview of all projects in the database

class, parent=None)


This class generates a html page listing the projects in the database

template file - Projects.html.tmpl

This can be a stand-alone page (including html and body tags) or included in another page.

To include this in another page set ‘full_html’ to ‘0’ in the args dict, and provide a css class for the table, ‘table_extra’ in the args dict.

default_order_by = 'ID'

generate HTML based on the requested dir, using the template file

getHtmlProjectTable(projects, table_id='project_table', table_extra='')

return the html table for the given projects


get the dictionary with the project - users


return the grouping of tables


return the top of the html page


(override) to return additional table row elements


(override) to return additional table headers

heading = '\n<link href="%(common_path)s/services/css/awe.css" rel="stylesheet" type="text/css">\n\n<!-- See webpage of jquery -->\n<script type="text/javascript" src="%(common_path)s/services/javascript/jquery.js"></script>\n<script type="text/javascript" src="${common_path}/services/javascript/jquery-ui.js"></script>\n<link type="text/css" rel="stylesheet" href="${common_path}/services/css/jquery-ui.css">\n\n<style type="text/css">\n/* definition for project table */\ntable {\n border-spacing: 2px;\n/* font-family : Verdana;*/\n}\ntable td {\n font-family : Verdana;\n font-size : 8pt;\n}\ntable th {\n background : #bbd;\n font-family : Verdana;\n font-size : 9pt;\n}\n</style>\n'

make html table with all users of the project

managers_in_tooltip = False
page_top = '\n<h2>Projects of %(db_name)s</h2>\n\n<p>\nTotal number of projects %(nr_projects)s<br>\nTotal number of users %(nr_users)s\n</p>\n%(project_click)s\n%(table_groups_html)s\n'
classmethod project_table_id(table_group='')

return the id of the project table if table_group is given, return the id of project group table

classmethod remove_from_url_query(url, remove_arg='project')

remove arg (default project) from the url query part

skip_project(project_id, project_name, description, instrument, privileges)

(override) return True to skip this project module

File Author WJ Vriend ( Date Dec 2004

class, template_string='')

Bases: object

Html Template class reads a template file and substitutes variables with values where variables can be placed inside repeating blocks

—– ===== Example 1 (Creating a template object) ===== —–

# from a file tmpl = HtmlTemplate(a_template_filename) # from a string tmpl = HtmlTemplate(None, template_string=”<html>…${VAR}…</html>”)

—– ===== Example 1 (filling in variables) ===== —–

[HTML Template]




tmpl = HtmlTemplate([HTML Template]) tmpl.setVariable( ‘VAR’, ‘Hello World’) # will replace ${VAR} with TEST tmpl.getHtml()

[HTML output]

Hello World


—– ===== Example 2 (doing blocks) ===== —–

[HTML Template]

<html><body> ${_BEGIN_BLOCK}


${_END_BLOCK} </body></html>


tmpl = HtmlTemplate([HTML Template]) a_block = tmpl.beginBlock(‘BLOCK’) a_block.setBlockVariable(‘VAR’, ‘Hello World 1’) a_block.updateBlock() a_block.setBlockVariable(‘VAR’, ‘Hello World 2’) a_block.updateBlock() a_block.setBlockVariable(‘VAR’, ‘Hello World 3’) a_block.updateBlock() tmpl.insertRepeatingBlock(a_block) tmpl.getHtml()

[HTML output]

Hello World 1 Hello World 2 Hello World 3


—– ===== Example 2 (doing blocks in blocks) ===== —–

[HTML Template]

<html><body> ${_BEGIN_BLOCK_OUTER}




${_END_BLOCK_OUTER} </body></html>


tmpl = HtmlTemplate([HTML Template]) a_outer_block = tmpl.beginBlock(‘BLOCK_OUTER’) a_outer_block.setBlockVariable(‘VAR_OUTER’, ‘Hello Outer World 1’) a_inner_block = a_outer_block.beginBlock(‘BLOCK_INNER’) a_inner_block.setBlockVariable(‘VAR_INNER’, ‘Hello Inner World 1’) a_inner_block.updateBlock() a_inner_block.setBlockVariable(‘VAR_INNER’, ‘Hello Inner World 2’) a_inner_block.updateBlock() a_outer_block.insertRepeatingBlock(a_block) a_outer_block.updateBlock() a_outer_block.setBlockVariable(‘VAR_OUTER’, ‘Hello Outer World 2’) a_inner_block = a_outer_block.beginBlock(‘BLOCK_INNER’) a_inner_block.setBlockVariable(‘VAR_INNER’, ‘Hello Inner World 3’) a_inner_block.updateBlock() a_outer_block.insertRepeatingBlock(a_block) a_outer_block.updateBlock() tmpl.insertRepeatingBlock(a_block) tmpl.getHtml()

[HTML output]

Hello Outer World 1
Hello Inner World 1 Hello Inner World 2
Hello Outer World 2
Hello Inner World 3



initialize a block


copy the filled blocks to the html file


generate the content from the template


return the content(html)

insertBlock(block_name, block_content)

insert block_content into the template where block_name


insert repeating block in template


open a template file and read content

replaceInContent(varName, varValue)

replaceInContent will replace the given string in the template content with the given value


save the the html to a file

setBlockValues(blockName, blockDicts)

setBlockValues will replace the variables in the template blockName with the given list of name,value pairs in blockDicts

setBlockVariable(varName, varValue)

set a variable within this block

setVariable(varName, varValue, overwrite=True)

setVariable will replace the variable in the template file with the given value

setVariableDict(varValuesDict, overwrite=True)

setVariableDict will replace the variables in the template file with the given values

setVariableList(varNames, varValues, overwrite=True)

setVariableList will replace the variables in the template file with the given values


update the block template

warn_on_not_used_vars = True
class, template)

Bases: object


initialize a repeating block

insertBlock(block_name, block_content)

insert block_content into the template where block_name


insert repeating block in template

setBlockVariable(varName, varValue)

set a variable within this block


copy the current filled block to the content


Bases: Exception

return begin block name as used in template

return end block name as used in template, BlockName)

get the begin and end position of block

return var name as used in template module




generate the html module

Collection of html (generation) utilities


Bases: object

EOL = '\n'
classmethod Escape(text)

escape html characters

classmethod UnEscape(text)

un-escape html characters

classmethod escapeOverlib(text)

overlib expects single quotes as ‘ and double quote as &quot;

classmethod generateOption(value, display, selected, disabled, extra='')

make a html <option value=..>..</option> and selecting or disabling the option

classmethod generatePrivateProjectSelect(sel_project, max_len=25, valid_instruments=[])

generate a html select box with all private projects

classmethod generateProjectSelect(privileges, sel_project, max_len=25, valid_instruments=[])

Generate html select box with projects with specified privileges

classmethod generatePublicProjectSelect(sel_project, max_len=25, valid_instruments=[])

generate a html select box with all public projects

classmethod generateSelect(name='', display=[], values=[], selected='', disabled=[], extras=[], select_name_values={})

make a html <select>…</select> from the display and values lists selecting the selected

classmethod generateSelectOptions(display=[], values=[], selected='', disabled=[], extras=[])

make a html collection of <option>…</option> from the display and values lists selecting the selected and disabling the disabled

classmethod make_radio_buttons(name, values, selected, displays=None)

Return html radio button definitions with name, one for each value . The radio button with value selected is selected. If displays is given then that will be shown as text, otherwise the values will be shown.

classmethod return_radio_checked(check, choice)

return radio choice module

Module with HttpBots class for bot detection

WJ Vriend


Bases: object

Class for testing if (http) client is a bot

isBot(ip, headers)

Is the given name a bot name ?

testValue(value, re_values)

Test all the re_names against the test name

useragents_file = 'HttpBots.cfg' module

Module with (example) Cleanup class for cleaning log files in the debug dir

WJ Vriend


Bases: object

(example) Class for cleaning up log files

Other Cleanup implementations must at least :
the __init__ method must accept the server object as argument the do_work method must be defined

remove old files


method which is called every X seconds


return the full log pathname

lock_name = 'Cleanup'
log_basename = 'CleanupThread.log'
long_log_age = datetime.timedelta(100)
short_log_age = datetime.timedelta(1) module

Class for handling the http(s) configuration

WJ Vriend

class, dict_type=<class 'collections.OrderedDict'>, allow_no_value=False, *, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section='DEFAULT', interpolation=<object object>, converters=<object object>)

Bases: configparser.RawConfigParser

ConfigParser that strips quotes from values

get(section, option, *args, **kwargs)
items(section, *args, **kwargs)

strip single or double from value


Bases: object


check the config dict


check the Environment


determine the configuration filename


return the configuration parser

getOption(option, default)

retrieve value of option, if not present use default

classmethod get_startup_args()

Return the startup arguments for the HTTP(S) server

classmethod import_class(class_specification)

import the class from the class specification, and return the python class


initialize the access_list from the config file

classmethod init_cleanup_thread(server)

init the cleanup (worker) thread specified in the config file (optional)


init the config struct from - the configuration file - the command line


initialize the host mapping dictionary


initialize the service list, dictionary coupling the html generation class to a http request


initialize the login Sessions


do all the configuration


Bases: Exception module

Database for the http server and services


Bases: object


initialize database tables

insert_query(query, args, duration, session=-1)

insert query row in database

tables = {'queries': (('created', 'timestamp'), ('query', 'text'), ('args', 'text'), ('username', 'text'), ('project', 'text'), ('privilegs', 'int'), ('http_session', 'int'), ('duration', 'real'))} module

module with HttpFileRequest class

WJ Vriend



Class for handling http file requests

content_types = {'.css': 'text/css', '.fits': 'application/fits', '.gif': 'image/gif', '.gz': 'application/octet-stream', '.htm': 'text/html', '.html': 'text/html', '.ico': 'image/x-icon', '.jpeg': 'image/jpeg', '.jpg': 'image/jpeg', '.js': 'application/x-javascript', '.log': 'text/plain', '.png': 'image/png', '.txt': 'text/plain', '.xml': 'text/xml'}

The http server serves files from either the root dir (/) or a virtual root (/home/user/static/) If virtual_root is specified in the dir_list section then we should only serve files from this root. If virtual_root is not specifed in the dir_list section then the access_list section should be used to limit the files being served. (though this mode should be considered obsolete) Path can be given in two manners; old and new style

new : [/]common/services/css/awe.css old : /full/path/to/repository/common/services/css/awe.css
default_content_type = 'application/octet-stream'

does the request exist and what is the absolute path


does the request exists and is allowed to serve ?

expire_delta = datetime.timedelta(1)

(derive) Derived classes can implement this method to do extra allowed checks


return headers for this file request


return the http content-type header


file is written directly to wfile, see write_wfile method


return the http status code

handler_database = False
http_date_format = '%a, %d %h %Y %T GMT'

is the request allowed to serve ?

log_enabled = False
readsize = 10000000

write message to the dump log


file is requested, read and return the content determine the filetype and return metadata of the file module

Module with the HttpSessionPool class


Bases: dict

Class for interacting with persistent data (serverside) for user sessions

The class itself is a dictionary storing per HtmlBase class a dictionary with session_ids pointing to a dictionary with session data :


The … can be used by the session to store data

Reserver keywords in this dict are :
date_started - when the session was started date_modified - last change in session

remove all the SessionPool objects older then the SessionLifeTime, the associated resources are removed by the objects cleanupUserSession method



getUserSession(cls, session_id)

get the global data for the class (cls) and user (ip,port)

initSession(ip, port, data)

initialize a new session and return new session id

lock_name = 'HttpSessionPool'
makeSessionId(ip, port, now)

generate session_id

setUserSession(cls, session_id, data)

store the global_data module

Module containing HttpStatistics class

class for handling statistics of http server

class, debug=False)

Bases: threading.Thread

Thread class listens on named pipe for statistic info, when the HTTP service is run in forked then the child processes use this to send statistics to the parent process.


run the thread

stop_command = 'STOP'
class, lock)

Bases: dict

addStat(stat, value, username, project)

add the value of the stat to the internal dict

appendStat(stat, value, username, project)

Append the value to the internal dict

corrupt_stats_mail = 'The statistics file "%s" is corrupt. Restore backup or statistics history is lost.'
day_date_format = '%Y-%m-%d'

init from old style stats


clean initialize

lock_name = 'HttpStatistics'
maxlen = 10

save the statistics to file

class, lock)

Bases: object

Class for collecting (simple) statistics, only count nr of GET, POST, files, etc …

addStat(stat, value=1)

Add the value to that statistic named stat, also add to the forever value

appendStat(stat, value)

Append the value to the statistic named stat

count_items = ['do_CONNECT', 'do_DELETE', 'do_GET', 'do_HEAD', 'do_OPTIONS', 'do_PATCH', 'do_POST', 'do_PUT', 'do_TRACE', 'request_invalid', 'request_not_existing', 'number_of_bytes', 'number_of_files', 'request_count']
getStat(stat, forever=False)

return the value for the given stat

ignore_items = ['forever_do_delete', 'forever_do_get', 'forever_do_head', 'forever_do_post', 'forever_do_put']
init_server_stat(stats, forever_items, stat_filename)

initialize the server stat file


initialize the statistics

lock_name = 'HttpStatisticsSimple'
maxlen = 10
newStat(stat, value)

create new stat with value


remove the given stat module

Http thread lock class


Bases: object

Dummy Locking class for the Http Server, this just raises an error when used


acquire the lock


release the lock


Bases: object

Thread Locking class for the Http Server


acquire the lock


release the lock module

Module with the HttpTimer class

class, worktime=None)

Bases: object

Class for doing timed actions in http server

  • does cleanup of user sessions in session_pool
  • updates the statistics file
  • mail the error log if errors occured in the last 24 hours
  • cleanup of log files
WorkTime = 300



method which is called every self.WorkTime seconds if argument last is True then stop after this call

log_filename = 'http_timer_error.log'
classmethod mail(from_address, to_address, subject, content, mail_server='')

send mail


mail the error log file in case of errors (once a day)


start the timer


write the forever stats to file


Bases: Exception module

Collection of sql methods for web services


Bases: object

SqlUtil class with sql helper methods for web services

classmethod pageSql(sql, rowmin, rowmax)

encapsulate query to enable paging (next page) see module

return an input file for wget to download multiple files/urls

WJ Vriend



Class for generating text file with multiple urls

input :

download_filename the filename of the script (default urls.txt)

download_urls list of pickled urls

< or >
download_files list of pickled filenames download_server the hostname of the server to download from

return additional header item(s)


return the content-type header


generate html module

Methods for text handling and formatting

WJ Vriend

translate value to human readable form : x KB, MB, GB, TB, etc … module

ThreadingHTTPServer is based on ThreadingTCPServer and ForkingHTTPServer is based on ForkingTCPServer.

The Threading class starts a new thread for each request (although multpiple request from one client are still done sequential ?), the forking class forks for each request.

Threading is very light in resources, forking heavy. But in case of threading the memory is shared between threads, which can result in seemingly random, very hard to find bugs.

Testing showed that the processing time for the threading and processing model (forking) is: - Threading 0.0002 to 0.0003 seconds - Processing 0.01 to 0.1 seconds

A simple html page was used as test object, the server and the client both run on the same machine.

WJ Vriend (

TODO - min / max amount of threads / processes - queuing of requests in existing threads / processes - get max_children and ALLOW_REUSE_ADDRESS from configuration

class, RequestHandlerClass, bind_and_activate=True)

Bases: socketserver.ForkingTCPServer

allow_reuse_address = 1

This overwrties the method collect_children in

If finished_only is set then only the finished (and defunct) processes are handled otherwise the original collect_children is called.

max_children = 100
class, RequestHandlerClass, bind_and_activate=True)

Bases: socketserver.ThreadingTCPServer

allow_reuse_address = 1

Dummy method, making the interface similar as ForkingHTTPServer

daemon_threads = True

Iterate all children, and discard the ones that are finished module, factor)

darken color by factor, g, b)

join the rgb components t oa color

split the given color in the rgb components module

File Author WJ Vriend ( Date Dec 2004

return object_id clause for use in sql

construct table name for sql module

File Author WJ Vriend wjvriend(at)astro(dot)rug(dot)nl Date Dec 2004

Python code for handling HTTP requests in a HTTP(S) Astro-WISE server

Configuration :

See HttpServer.cfg for the configuration options, this configuration file (global) should exist in the same directory as the current file. With the -c option a custom configuration file can be specified.

See class myHttpRequestHandler for more information

class, client_address, server)

Bases: http.server.BaseHTTPRequestHandler


Handles post, get and head requests as a Http server.

myHttpRequestHandler class is created by BaseHTTPServer (http_server) everytime a http request is made.

For each request 1. a class is instantiated with the given parameters 2a. the class method getContentType is called for the content-type 2b. the class method getAdditionalHeaders is called for optional headers 2c. the class method getCookies is called for optional cookies 3. the class method getHtml() is called to generate the html page

Static file requests are handled by the http server itself.

The http server has three toggles: -verbose For (very) verbose output (TODO; couple this to the log object) -performance For every request the processing time is logged -sql Logs every sql statement (TODO; couple this to the DB layer)

The following parameters will be used by the server for fillign the Profile : - project The project to be set in the Profile - privileges The privileges to be set in the Profile - database_name The database name (or connection string) to use - username The username for making the database connection

The following parameters are always set by the http server in the args dictionary: * _maintenance_lines * _request * _request_method * _http_error

The html generation classes can specify the following attributes : - handler_database create a database connection for this request ?

For storing data between sessions the session_pool can be used.

The server keeps track of statistical information since last start and forever.

The coupling of request and class is hard-coded, see the service_list for the coupling. The service list is defined in the configuration file. If enabled then a status page of the web server can be requested using the url “http://server:port/Config”.

myHttpServer startup syntax:

>awe $AWEPIPE/common/services/general/myHttpServer

-c configration file
-h hostname
-p port is the port number the server listens on
-d domain is the domain the server is part of (example this is used for the cookies
-s default_service is one of the above stated requests. This will be be served when nothing speicific (or /) is requested.
-t test mode, dump all logging on the screen
-v (set) version

If an non empty file called ‘maintenance’ exists in the root of the web server then the contents of this file will be showed for every request.


main method for handling a request

addStat(stat, value=1)

Add the value to that statistic named stat, also add to the forever value

appendStat(stat, value)

Append the value to the statistic named stat

classmethod cleanLogLine(line)

Clean the log line, see cleanLogLineArg()

classmethod cleanLogLineArg(arg)

Clean the log line argument, remove password entries


stop close http fd’s, DB, stop logging, remove profile

classmethod cls_getCookieName(cookie_name, server_name, old_style=False)

Return the cookie name to use, prepended with the server name Server name and cookie were joined with “_”, from 9 Feb “#” is used (so server name can contain “_”)


try to connect to the database, using the given profile


try to connect, log if failure

cookie_aw_session_expires = 1
cookie_max_size = 4000
cookie_sso_expires = 24

handle CONNECT request, is same as do_GET ? (not tested)


handle DELETE request, is same as do_GET ? (not tested)


handle a GET request


handle a HEAD request, same as GET but only returning headers


handle OPTIONS request, is same as do_GET ? (not tested)


handle PATCH request, is same as do_GET ? (not tested)


handle a POST request


handle PUT request, store PUT information in _put_file as file object


handle TRACE request, is same as do_GET ? (not tested)


Called by BaseRequestHandler when request is finished

classmethod form_to_dict(form_data)

put form data in dict, form elements with name can have multiple values


Format the error message, this is called in case of unhandled expection, In case of custom error call this method to format the error message


return the number of active threads or processes


determine and return the IP address of the client the IP can be obscured by a proxy

getCookieName(name, old_style=False)

See class method version; cls_getCookieName


If cookie with name exists return value, otherwise None Cookie names are stored with the server name prepended


return the cookies meant for given service


Return the data dictionary prepared for log


Return the object handling this request


Return the class handling this request

getStat(stat, forever=False)

return the value for the given stat


Return the full url of this request


Return the full url of this; protocol, server, path

getUserSession(session_id, session_class=None)

return the global (persistent) data for the request


start of every log line


fill self.cookies dict with all cookies meant for this service


handle the input arguments dictionary, decode the binary values to string


handle exceptions, if user cancels request, log it and continue, else format exception in html


initialize for the request, there can be multiple requests handled by one request handler instance


initialize the request path; unquote and possibly add path


set the profile; variables username, password, project


write error message to the error log file

log_message(format, *args)

called from BaseHTTPRequestHandler, logging method for main thread, some (boring) log is writtten to dump file the rest is printed, use Message() to log thread specifics


make a cookie string from either a list/tuple or dict (preferred)

newStat(stat, value)

create new stat with value


remove the given stat


send the http headers

setUserSession(global_data, session_id=None, lock=True, session_class=None)

set the global (persistent) data for the request


start logging for this request

status_codes_ok = (200, 201, 301, 302, 303, 307, 308)
tryProcess(request, method)

try processing the request


place the name value pairs from the cookies in the data dictionary and extract the username and password from the possible login cookies

writePipe(action, stat, value, username, project)

write the stat and value on the named pipeline


Bases: Exception module

File Author WJ Vriend wjvriend(at)astro(dot)rug(dot)nl Date Dec 2004

Python code for running an HTTP(s) server in the Astro-Wise environment

Configuration :

See HttpServer.cfg for the configuration options, this configuration file (global) should exist in the same directory as the current file. With the -c option a custom configuration file can be specified.

Enabling HTTPS :

* Certificates *

When using ssl a private key and certificate are required, these are the steps to create them :

openssl genrsa -des3 -out server.key 1024 openssl req -new -key server.key -out server.csr openssl rsa -in server.key -out server.key openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

server.crt is the certificate, server.key is the private key see $AWEPIPE/common/services/genereal/HttpServer.cfg for an example of the configuration file under the [ssl] section the key and certificate must be specified

* Keys *

The login service uses the following keys (in Env):

encryption_key; to encrpyt and decrypt the username/passwords login_server_key_init; to allow other services to initiate a Session login_server_key_request; to allow other services to request a Session

Example how to generate random base64 encoded keys # python code import uuid u = uuid.uuid4() u.bytes.encode(“base64”)

main method module

Original from cgi.parse_multipart

Added report of upload progress back to handler

To make use of the upload progress provide the following arguments in the POST or GET session:
‘_upload_classname’ and ‘_upload_id’
Then in an UserSession with id ‘_upload_id’ the following information is stored :
‘content-length’ : the total content length (from the header) ‘uploaded’ : the number of bytes uploaded

At the end the ‘uploaded’ will be set to ‘content-length’ (otherwise the ‘uploaded’ will never be equal to ‘content-length’,

because we can not determine the header length), pdict, handler, content_len, header_len=0)

Module contents