common.toolbox package

Submodules

common.toolbox.DScompressCheck module

common.toolbox.DScompressCheck.main()
common.toolbox.DScompressCheck.visit(arg, dir, names)

common.toolbox.TableStats module

common.toolbox.TableStats.AIndexALID(c, table_name, ind, attr_list, owner)
common.toolbox.TableStats.AIndexALIDAID(c, table_name, ind, attr_list, owner)
common.toolbox.TableStats.AIndexSLIDSID(c, table_name, ind, attr_list, owner)
common.toolbox.TableStats.CheckAIndexALID(c, table_name, ind, attr_list, owner)
common.toolbox.TableStats.CheckAIndexALIDAID(c, table_name, ind, attr_list, owner)
common.toolbox.TableStats.CheckSIndexSLIDSID(c, table_name, ind, attr_list, owner)
common.toolbox.TableStats.ParCheckALIDAID(c, table_name)
common.toolbox.TableStats.SIndexDECSLID(c, table_name, ind, attr_list, owner)
common.toolbox.TableStats.SIndexHTM(c, table_name, ind, attr_list, owner)
common.toolbox.TableStats.SIndexSLIDSID(c, table_name, ind, attr_list, owner)
common.toolbox.TableStats.doit(c, q, exe)
common.toolbox.TableStats.doitlock(c, q, exe, table_name, owner, uselock=True)
common.toolbox.TableStats.get_tablespace(c, table_name, owner)
common.toolbox.TableStats.lock(table_name, owner)
common.toolbox.TableStats.lockfile(table_name)
common.toolbox.TableStats.main()
common.toolbox.TableStats.mygetpass(*args)
common.toolbox.TableStats.unlock(table_name, owner)
common.toolbox.TableStats.unlockfile(table_name)

common.toolbox.VacuumObjectLists module

common.toolbox.compile_pipeline module

common.toolbox.dbaddprojectPostgresSQL module

common.toolbox.dbaddprojectusers module

common.toolbox.dbaddprojectusersPostgreSQL module

common.toolbox.dbaddtablestostreams module

common.toolbox.dbaddtablestostreams.awglobalname(infix)
common.toolbox.dbaddtablestostreams.awinfix(infix)
common.toolbox.dbaddtablestostreams.maintain_tables(hub, spokes, tablenames)
common.toolbox.dbaddtablestostreams.set_table_instantiation_scn_for_hub(spokes, tablenames)
common.toolbox.dbaddtablestostreams.set_table_instantiation_scn_for_spokes(hub, tablenames)

common.toolbox.dbawextrasourcelisttable module

common.toolbox.dbawoper module

common.toolbox.dbawoperPostgreSql module

common.toolbox.dbawpartition module

common.toolbox.dbawutil module

common.toolbox.dbdpu module

common.toolbox.dbflatremake module

common.toolbox.dbflatremake.compile_view(table_name)
common.toolbox.dbflatremake.drop_type(table_name)
common.toolbox.dbflatremake.get_typename(table_name)
common.toolbox.dbflatremake.get_viewname(argument)
common.toolbox.dbflatremake.grant_view(table_name)
common.toolbox.dbflatremake.make_declare_typedefinition(table_name, columndefs)
common.toolbox.dbflatremake.make_declare_typename(table_name)
common.toolbox.dbflatremake.make_inline_views(table_name, columndefs)
common.toolbox.dbflatremake.make_view(table_name, columndefs, baseview=False)

common.toolbox.dbflatremakePostgreSql module

common.toolbox.dbgrants module

common.toolbox.dbgrantsPostgreSql module

common.toolbox.dbimportall module

common.toolbox.dbmigrateall module

common.toolbox.dbnewpassword module

common.toolbox.dbnewuser module

common.toolbox.dbnewuserPostgreSQL module

common.toolbox.dbshowproject module

common.toolbox.dbvacuum module

Tool to remove objects from the database and files on the dataserver.

SYNOPSIS

awe dbvacuum.py <gpgkeyid>

DESCRIPTION Users delete their objects from MyDB. This means that they are flagged as deleted. Although the objects are no longer visible to users they - and any attached files - still exist. This tool removes all objects from the database _after_ ensuring that the corresponding files are removed from the dataserver. If a file that belongs to an object cannot be removed from a dataserver, the database object will not be removed. This can happen when a dataserver cannot be reached, for example. Yet, during a next run, when the dataserver can be accessed again, the file and the data object will be removed.

The <gpgkeyid> is the id of the gpg key of the account running this tool. The public key for this id needs to be one of the trusted keys for the account that runs a dataserver. The dataserver only accepts delete requests that are signed with this trusted key. To list the id’s of the keys, do

gpg –list-keys

Note that those database objects, that refer to a file, will not be removed if _any_ of the dataservers cannot be accessed. So, to be able to remove such a database object, _all_ dataservers must be accessible.

class common.toolbox.dbvacuum.DataFile(resultstring)

Bases: object

delete()
setattribute(item)
common.toolbox.dbvacuum.delete_all_objects_from_table(cursor, table_name)

Remove all objects from a table that does not refer to a file.

common.toolbox.dbvacuum.delete_file_from_dataserver(filename)

Remove a file on all dataservers where it exists. Return True if this succeeds and False if a file could not be removed.

common.toolbox.dbvacuum.delete_object_from_database(cursor, table_name, object_id)

Remove given database object.

common.toolbox.dbvacuum.delete_objects_without_file_from_table(cursor, table_name, file_attribute)

Remove all objects from a table that do not refer to a file.

common.toolbox.dbvacuum.get_files_to_delete(cursor, table_name, file_attribute)

Return object_id’s and filenames for all deleted objects in a table.

common.toolbox.dbvacuum.get_tables_with_column(cursor, column_names=())

Return all TABLEs (not VIEWs) with a given columnname.

common.toolbox.dbvacuum.locate_with_all_suffixes(filename)
common.toolbox.dbvacuum.number_of_objects_with_filename(cursor, filename, tables_with_filename)

Return the number of non-deleted objects with a given filename.

common.toolbox.distutils_README module

template = {
‘package_name’ : ‘’, ‘package_author’ : ‘’, ‘author_email’ : ‘’, ‘init_env_python’ : ‘’, ‘package_help_url’ : ‘’, ‘example_output’ : ‘’ ‘examples’ : ‘’,

}

And then merge wth :

python common/toolbox/README_common.py README_specific.py

common.toolbox.distutils_setup module

# setup.py project specific example file # # imports from distutils.core import setup from common.toolbox.distutils_setup import architecture, InstallOracle, check_environment, get_packages, get_package_data, get_data_files # # check environment, extend with project specific checks check_environment() # # call setup setup(name = <package_name>,

version = <package_version>, description = <short_description>, long_description = <long_description>, author = <person_name>, author_email = <person_email>, maintainer = <person_name>, maintainer_email = <person_email>, url = <documentation_url>, download_url = <download_url>, platforms = [‘Linux’, ‘Darwin’, ‘Windows’], license = <license>, #requires = [<required_packages>], # note that these are not enforced during install (OBSOLETE) install_requires = get_install_requires(<path_to_requirements.txt>), # or provide a list of required packages packages = get_packages([‘common’, <project>]), package_data = get_package_data([‘common’, <project>]), #include_package_data = True, # not yet supported ? #package_dir = {‘’: ‘awe’}, # not sure what to do with this data_files = get_data_files(), cmdclass = { ‘install_oracle’: InstallOracle, },

)

class common.toolbox.distutils_setup.InstallOracle(dist)

Bases: distutils.cmd.Command

distutils class for installing Oracle instant client and cx_Oracle

cx_oracle_dir = 'cx_Oracle-5.1.2'
cx_oracle_url = 'http://astro-wise.org/awesoft/cx-oracle/cx_Oracle-5.1.2.tar.gz'
description = 'Installs the Oracle instant-client and cx_Oracle package. If cx_Oracle is already installed nothing is done.'
finalize_options()

make user input directories absolute

initialize_options()
install_cx_oracle(instantclient_path)

install cx_oracle

install_oracle_client(install_dir)

install the Oracle client

install_oracle_connection(install_dir)

install oracle instant client and cx_Oracle

instantclient_dir = 'instantclient_11_2'
instantclient_urls = {'Darwin': {32: ('http://astro-wise.org/instantclient/instantclient-basic-macos.x32-11.2.0.3.0.zip', 'http://astro-wise.org/instantclient/instantclient-sdk-macos.x32-11.2.0.3.0.zip', 'http://astro-wise.org/instantclient/instantclient-sqlplus-macos.x32-11.2.0.3.0.zip'), 64: ('http://astro-wise.org/instantclient/instantclient-basic-macos.x64-11.2.0.3.0.zip', 'http://astro-wise.org/instantclient/instantclient-sdk-macos.x64-11.2.0.3.0.zip', 'http://astro-wise.org/instantclient/instantclient-sqlplus-macos.x64-11.2.0.3.0.zip')}, 'Linux': {32: ('http://astro-wise.org/instantclient/instantclient-basic-linux-11.2.0.3.0.zip', 'http://astro-wise.org/instantclient/instantclient-sdk-linux-11.2.0.3.0.zip', 'http://astro-wise.org/instantclient/instantclient-sqlplus-linux32-11.2.0.2.0.zip'), 64: ('http://astro-wise.org/instantclient/instantclient-basic-linux.x64-11.2.0.3.0.zip', 'http://astro-wise.org/instantclient/instantclient-sdk-linux.x64-11.2.0.3.0.zip', 'http://astro-wise.org/instantclient/instantclient-sqlplus-linux.x64-11.2.0.3.0.zip')}}
run()
test_cx_Oracle()

test is cx_Oracle is installed / can be imported

user_options = [('home=', None, 'see install command'), ('prefix=', None, 'see install command'), ('exec-prefix=', None, 'see install command'), ('user', None, 'see install command')]
common.toolbox.distutils_setup.check_environment()

check the environment

common.toolbox.distutils_setup.chown_if_needed(path)

if the script is run with sudo, change the ownership of the path to the original userid (uid) and groupid (gid)

common.toolbox.distutils_setup.get_data_files()

The data files to handle

common.toolbox.distutils_setup.get_home_dir()

return the home dir of the (sudo) user

common.toolbox.distutils_setup.get_install_requires(path_to_requirements)

read the requirements.txt file and return the requirements as list

common.toolbox.distutils_setup.get_package_data(input_directories)

return package data, all non python files in given directories

common.toolbox.distutils_setup.get_packages(dirnames)

return packages, all python files in given directories

common.toolbox.dpu_server_dump_file module

Script for handling the DPU dump file

common.toolbox.dpu_server_dump_file.dpu_dump_clean(dpu_dump_file)

make a copy of the dpu dump file and write clean version

common.toolbox.dpu_server_dump_file.dpu_dump_view_compact(dump_file)

print the dpu dump file compact

common.toolbox.dpu_server_dump_file.dpu_dump_view_full(dump_file)

print the dpu dump file full

common.toolbox.dpu_server_dump_file.main()

main method, parse options and execute

common.toolbox.gitversioning module

gitversioning.py

Script to put git version nummers in python “__version__” variable.

Type <your python executable> <path-to-girversioning.py> to get help.

common.toolbox.gitversioning.get_fileversionnumber(codefile)
common.toolbox.gitversioning.get_gitversionnumber(codefile)
common.toolbox.gitversioning.put_versioninfile(codefile, version)

common.toolbox.update_cvs module

common.toolbox.update_git module

common.toolbox.update_pipeline module

common.toolbox.versioning module

This script can be run on an awe checkout directory to compare it with the version tagged as AWBASE. This script will create a files such as

astro_main_BiasFrame.cvsdiff

which describes the path to the file from the base of the checkout directory with slashes replaced with underscores, and contains the cvs diff output for that file.

Example usage: > python versioning.py ../aweTEST

common.toolbox.versioning.get_logs_for_cvs_diff(filename, cvspath, cvstag)
common.toolbox.versioning.interpret_cvs_diff_stderr(redirected_error_file, cvspath, cvstag)
common.toolbox.versioning.interpret_cvs_diff_stdout(redirected_output_file, cvspath, cvstag)
common.toolbox.versioning.main()

Module contents

This directory contains a set of tools.