# HOW-TO Use SAMP in Astro-WISE¶

The Simple Application Messaging Protocol (SAMP) is a protocol for astronomical applications to collaborate. A SAMP client is available from the awe-prompt which allows interaction with visualization software such as Topcat and Aladin. Furthermore, a set of new SAMP messages has been designed to allow interactive query driven visualization through data pulling.

The idea behind SAMP is akin to the UNIX-philosophy that tools should do one thing, should do that thing well and communicate with other programs for things they cannot do. E.g.: A specific piece of software is responsible for retrieving the data from a data source, another program computes parameters which then get visualized with a third program.

The general principle in SAMP is that there is a central HUB, to which clients connect. The clients send messages through the HUB to other clients, which (can) respond with the result of the requested actions. There is a lot of freedom in what messages can be send, and the protocol is designed to be language agnostic. SAMP can be seen as the successor to PLASTIC (PLatform of AStronomical Tool Interaction) which had a similar goal, but never became a VO standard.

## SAMP HUB and Clients¶

Figure 1 shows a diagram of the interoperability between Astro-WISE and SAMP.

### SAMP HUB¶

The center of the SAMP protocol is the SAMP HUB. The principal function of the HUB is to register connected applications and to relay messages inbetween them. The HUB itself does not have to have any human interface at all, let alone a GUI. The HUB is often integrated in one of the clients, e.g. Aladin and Topcat, but standalone HUBs exist as well.

### Topcat¶

Topcat [1] is a table viewer/manipulator written in java. It can read a large variety of tabular data, from FITS and VOTable to common csv files. It is now developed by AstroGrid. Topcat has some nice visualization options, such as 2D or 3D scatter plots and primitive density plots. The power of the visualizations lies in the interactivity. Select points in one scatter plot, and they become automatically highlighted in another plot which might show entirely different parameters of the data points.

Aladin [2] is an FITS image viewer developed in France by the Centre de Donn’ees astronomiques de Strasbourgh (CDS). It can view local files, connect directly to several image repositories, by the VO and otherwise. Furthermore it can receive images from other applications such as the VO Desktop through SAMP. Aladinis developed in java and can load images up to 50K by 50K pixels.

## SAMP Astro-WISE integration¶

The awe-prompt includes a SAMP client as a Python module. This allows an astronomer to combine the large scale data handling from Astro-WISE with the visualization tools from other SAMP applications.

### Example SAMP usage in AWE¶

A SAMP HUB needs to be started first in order to use SAMP at all, we recommend the one in Aladin or Topcat. Besides a HUB, other SAMP enabled applications (such as Topcat or the VO Desktop) should be started as well. Start SAMP from the awe-prompt as follows:

awe> from astro.services.samp.Samp import Samp
awe> samp = Samp()


One of the most common usages of SAMP is to broadcast the data of objects to inspect them with other applications:

awe> sls = (SourceList.OBJECT == '2df_V_18') & \
.... (SourceList.creation_date > datetime.datetime(2007, 7, 1))
awe> sl = sls.min('creation_date')
awe> print(sl)
Name of SourceList : SL-HBUDDELMEIJER-0000135591
SourceList ID      : 135591
Sources in list    : 3226
Parameters in list : 35

[newton] 12:33:28 - Retrieving Sci-GSIKKEMA-WFI-------#843--...


This will show the frame in Aladin with the source catalog overlaid. Topcat will have loaded the SourceList as a table.

The visualization software can be used to to select interesting sources for further investigation in the awe-prompt. With Aladin this is done by either hoovering over a source (highlighting it), or by dragging a rectangle around a group of sources (selecting them). Other software such as Topcat have similar mechanisms. The highlighted or selected sources can be requested on the awe-prompt.

awe> samp.highlightedSource(sl)
(135591, 1635)
awe> samp.selectedSources(sl)
[(135591, 1710), (135591, 1773), (135591, 1787)]


These SIDs can now be used for further processing, e.g. with GalFit.

awe> sids = [ids[1] for ids in samp.selectedSources(sl)]
awe> dpu.run('GalFit', slid=sl.SLID, sids=sids)


### Communication with HUB¶

Connecting and registering with the HUB is done automatically when the Samp class is instantiated. This can be suppressed with the register=False parameter. The following procedure registers manually:

awe> from astro.services.samp.Samp import Samp
awe> # instantiate the class, this also starts an XML-RPC server
awe> # in the background to receive messages from the HUB
awe> samp = Samp(register=False)
awe> # read the ~/.samp file for settings of the hub
awe> settingsHub = samp.getSettingsHub()
awe> for k in settingsHub:
....     print('%-20s %1s'%(k,settingsHub[k]))
....
samp.hub.xmlrpc.url  http://127.0.0.1:59102/xmlrpc
samp.secret          7ce936d9b5fea215
hub.start.date       Fri Jun 19 15:16:21 CEST 2009
samp.profile.version 1.1
hub.impl             org.astrogrid.samp.hub.BasicHubService
awe> # connect to the XML-RPC server provided by the HUB
awe> XmlRpcHub = samp.connect()
awe> # register us with the HUB and declare our message subscriptions
awe> infoRegistration = samp.register()
awe> for k in infoRegistration:
....     print('%-20s %1s'%(k,infoRegistration[k]))
....
samp.self-id         c3
samp.private-key     k:4_wavkccristijjefj
samp.hub-id          hub


Information about (other) registered clients can be requested with getClients. Information other clients can see from the Astro-WISE class is stored in metadata.

awe> clients = samp.getClients()
awe> for cid in clients:
....     print(cid,clients[cid]['samp.name'])
....
c2 topcat
hub Hub
awe> for k,v in clients['c1'].iteritems():
....     print('%-24s %s'%(k,v))
....
author.email             fernique@astro.u-strasbg.fr
author.name              Pierre Fernique, Thomas Boch
author.affiliation       CDS, Observatoire astronomique de Strasbourg
samp.description.text    The Aladin sky atlas and VO Portal
....     print('%-24s %s'%(k,v))
....
author.email             buddel@astro.rug.nl
author.name              Hugo Buddelmeijer
home.page                http://www.astro-wise.org
author.affiliation       Kapteyn Astronomical Institute, Groningen
samp.name                Astro-WISE
samp.description.html    <p>Astro-WISE</p>
samp.documentation.url   http://www.astro-wise.org
samp.icon.url            http://www.astro-wise.org/pics/logo-samp-astrowise.png
samp.description.text    Astro-WISE.


### Sending Tables and Images¶

Sending data is most easily achieved with one of the broadcast functions. The broadcast functions send data to all clients (that can handle that datatype).

• broadcastSourceList(sourcelist, filename=None, tableid=None, name=None): converts sourcelist to an VOTable and uses broadcastVOTable to send it.
• broadcastSourceCollection(sourcecollection, filename=None, ..): converts sourcecollection to an VOTable and uses broadcastVOTable to send it.
• broadcastCatalog(catalog, filename=None, tableid=None, ..): converts catalog (a PhotSrcCatalog) to an VOTable and uses broadcastVOTable to send it.
• broadcastTableConverter(tableconverter, filename=None, tableid=None, name=None): converts tableconverter to an VOTable and uses broadcastVOTable to send it.
• broadcastVOTable(filename, tableid=None, name=None): Sends a VOTable using the table.load.votable message.
• broadcastFrame(frame, filename=None): Downloads the frame (as FITS) and sends it using broadcastImage.
• broadcastImage(filename): Broadcasts a FITS image.
• broadcast: This function accepts an object that is either a SourceList, a SourceCollection, a Frame, a PhotSrcCatalog or a TableConverter and dispatches it to the relevant function.

The filename, tableid and name keywords are respectively the filename where the (intermediate) VOTable or FITS file is saved, an identifier for the table and a name to display in the other programs.

At the time of writing, the table.load.fits message is not fully supported in all SAMP applications, therefore all broadcast functions send only VOTables. The (tabular) data is converted to a ‘TABLEDATA’ votable. All broadcasted files (VOTables and fits images) are stored locally on disk, of which the location is broadcast through SAMP.

### Sending Interaction Messages¶

Interactivity between different SAMP programs for tabular data can be achieved with the highlight function (which sends the table.highlight.row message) and the select function (which sends the table.select.rowList message). Their first argument is a ‘table key’ (see section Storing Information), the second one SID or a list of SIDs respectively.

awe> samp.highlight(sl, 1234)
awe> samp.select(sl, range(1000,1100))


The pointAtSky function uses the coord.pointAt.sky message to let other applications point to a certain position on the sky. E.g., Aladin will center the active image plane on the position.

awe> samp.pointAtSky(123.0, -20)


### Sending General Messages¶

SAMP describes 4 methods to send messages (of any type) which are mapped to similar function names which can be used to send messages manually. This is usefull to send messages that are not supported by the class (yet). Since the message types are agreed upon between individual applications, any application developer (you) can create their own messages. All the above commands use the callAll method to send their requests.

• notify(receiverid, message): Sends the message to one specific receiver, it is not possible to reply.
• call(receiverid, message): Sends the message to one specific receiver. The receiver is expected to reply, but the program (the awe-prompt) continues.
• callAll(message): Sends the message to all clients that have registered for this MType. The receivers are expected to reply, but the program (the awe-prompt) continues.
• callAndWait(receiverid, message, timeout=20): Sends the message to one specific receiver. the program (the awe-prompt) waits at most timeout seconds on the recipient to reply.

The example below sends an Aladin script with several methods (it is assumed client ‘c1’ is Aladin). In the case of call it is likely that only Aladin will be registered to this mtype and thus all example will achieve the same thing. (Sending scripts through SAMP is an undocumented feature of Aladin).

awe> message = {
....   'samp.params':{'script':'zoom 16x'}
.... }
awe> samp.callAll(message)
awe> samp.call('c1',message)


The receiverid parameter is the id of the receiver. These can be found with the getClients method. For the Aladin hub these are just the letter c followed by a number. For convenience all above function use the fixReceiverID which allow the name of the client (as found through getClients) as the receiverid. For Topcat and Aladin it is also possible to use the shortcuts topcat and aladin as receiverid.

The notify command does not allow replies, but it is implemented in the same way as the call method, so replies that are send anyway are still stored. The callAndWait functionality is mainly available for non multithreading applications. It can also be useful if it is essential to wait for the answer, so it is implemented in the Python class as well. Some applications (such as Aladin) do not always reply, so make sure to set a timeout.

### Receiving Messages¶

Tables can be send to the awe-prompt with the table.load.votable message. In Topcat this can be done by broadcasting a table or a subset. In Aladin by broadcasting the relevant plane. The table itself is only stored if the load_tables member of the SAMP instance is set to True.

For interactivity for tables, the Python class can receive the table.highlight.row and table.select.rowList. The chosen sources are stored in the tables member, see section Storing Information.

In Aladin highlighting a source is done by hoovering over a source, in Topcat this can be done by setting ‘Activation Action’ to ‘Transmit Row’ and selecting a row. Selecting multiple sources in Aladin is done by dragging a green rectangle around the sources, this is automatically broadcast with the table.select.rowlist message. In Topcat it is possible to send subsets as selections either when creating the subset or later from the ‘subsets’ window.

Retrieving highlighted and selected sources can be done with similarly named functions

• highlighted(tableid): Returns the latest highlighted source from table tableid as an integer row number (SID in the case of a SourceList).
• selected(tableid): Returns the latest selected sources from table tableid as a list of integer row numbers (SIDs in the case of a SourceList).
• highlightedSource(tableid): Returns the SLID-SID combination of the latest highlighted source.
• selectedSources(tableid): Returns the SLID-SID combinations of the latest selected sources.

### Storing Information¶

Of every table that is send through SAMP, the class records its properties in the table member.

awe> for k,v in samp.tables[sl].iteritems():
....     print("%-12s %s"%(k,v))
....
name         SourceList-135591
url          file://localhost/Users/users/buddel/SL-135591-2df_V_18.votable
tableid      SourceList-135591
type         votable
selected     [1710, 1773, 1787]
highlighted  1635


The url, tableid and name are described in the table.load.votable message type. They are respectively the location of the intermediary votable, the id to be used within SAMP and the name to be displayed in the applications. The type denotes the type, at this moment only votable is supported.

The highlighted key stores the id of the last source that has been highlighted with the table.highlight.row message as an integer. The selected key stores the ids of the last set of sources that have been selected through the table.select.rowList message as a list of integers. In the case of SourceLists, the ids are the same as the SIDs.

The primary key of the tables dictionary is the SAMP table-id which is stored in the tableid key. For Python objects (SourceList, PhotSrcCatatalog, TableConverter), the object itself can be used as key as well. For SourceLists, the SLID (as integer) is also a key.

All messages that are send are stored in the messagesSend member, including with possible replies. All received messages are stored in messagesReceived. The messages are stored as a dictionary, of which the actual message is stored as an item. An example of a message in messagesSend:

{
'messageTag': 2,
'messageId': {'c1': 'c3_A_16a8_2'},
'type': 'callAll'}
'time': 1245661731.7381041,
'message': {
'samp.params': {
'url': 'file://localhost/[..]/SL-135591-2df_V_18.votable',
'table-id': 'file://localhost/[..]/SL-135591-2df_V_18.votable',
'name': 'SourceList-135591'
},
},
'replies': {
'c1': {'samp.status': 'samp.ok', 'samp.result': {}},
},
}


messageTag is an unique identifier for our client (consecutive integers starting from 1), messageId is the unique id of the message given to it by the HUB (it the return value for the call methods). The type denotes how the message is send, broadcasted to all clients in this case, receiverId denotes who got the message, everyone in this case. time is the value returned by time.time() on sending the message (“current time in seconds since the Epoch”). The actual message is stored with key message. The replies are stored in replies which is a dictionary with as keys the SAMP client-id (c1), the client name ('Aladin') and in some cases shortcut names ('aladin'). Note that only the client-id is always unique.

## Query Driven Visualization through SAMP¶

The awe-prompt SAMP client accepts several new message types to pull data and to inspect and influence its derivation over SAMP. The paradigm of Target Processing includes data pulling with full data lineage, which can be abstracted well. These messages could therefore easily be used in other information systems as well and are described here in a general form, highlighting the Astro-WISE specifics.

Within Astro-WISE, the messages are currently only applicable to SourceCollections. In the SourceCollection HOW-TO a set of prototype applications that use these messages are shown.

## Data Pulling Messages¶

There are two new messages to pull catalog data out of the information system.

• catalog.pull: Pull a catalog and send it over SAMP using one of the table.load.* messages. This message requires the following parameters, detailed below: an identifier of a catalog to select the sources from, a selection criterion and a list of requested attributes of the sources. The awe-prompt SAMP cliet will create a dependency tree of SourceCollections whose end node contains the requested catalog.
• catalog.derive: Perform the same action as catalog.pull, but without sending the catalog data over SAMP.

The .derive message is useful when it is necessary to inspect or modify the derivation of the catalog—using the messages in section Object Messages before visualization. These two messages require three parameters which we should elaborate on:

• catalog-id: An identifier of the base catalog to select the sources from. For the Astro-WISE SAMP client this has to be the SCID of a SourceCollection. This could be extended in the future, for example by referring to an observation.
• query: A selection criterion to specify which sources of the original catalog are requested. This should be a logical expression referencing the attributes below. For the Astro-WISE SAMP client this should be a string that is suitable for use in a FilterSources SourceCollection.
• attributes: A list of requested attributes (parameters) of the sources. It is not required that the catalog corresponding to the table-id contains these attributes. The attributes should be specified as a comma-separated list of attribute names for the Astro-WISE SAMP client.

## Object Messages¶

Several SAMP message types are defined for interaction with an information system that store data through persistent objects. These messages allow the visualization software to gain information about the objects and inspect or even influence its processing. Although the messages are designed to be applicable for any object, they are currently only supported for SourceCollections. The persistent object related message are:

• object.highlight: Highlight an object.
• object.info: Return information about an object, see below.
• object.change: Change the value of a property of an object such as a process parameter or a dependency.
• object.action: Perform an action related to an object or property. Possible actions are retrieved using the object.info message.

The object.highlight message can be send to any application, the others are supposed to be send to the information system only.

### SAMP object.info Data Structure¶

A specific SAMP map is defined as a return value for the object.info message, containing information about the object and its properties. This is generated by the get_export() function of the Astro-WISE classes, which is currently only implemented for SourceCollections. For the object itself it includes information about its processing status, whether the object can be modified and what properties it has.

The properties of an object include process parameters and references to the progenitors of the object. The returned information of a property include its name, current value and optionally other values it can be set to. Furthermore the information system can define actions that can be performed on the object or its properties.

## More and future features¶

In the subscriptions member it is stored which SAMP message type gets mapped to which function. By updating this dictionary before registering with the HUB, it is possible to hook your own functions to specific types. Perhaps a better hooking mechanism would be useful.

The class can also be used standalone, then it can act as a proxy for non-interactive clients. These clients can connect to the XML-RPC server to request which sources are highlighted by other applications and such. There is only preliminary support for this now.

Currently, only one highlighted row and one selected row list is stored. Perhaps in the future highlighted and selected sources will be stored per SAMP client.

## SAMP Protocol¶

SAMP is a simple but extensible protocol, using a client-server model based on application defined messages. We give a short description of the protocol to the extend that suffices how the protocol works from the user perspective. For full details, refer to the official documentation

[3] or the official wiki [4].

Clients register with the SAMP hub and register for certain types of messages. Clients can than send messages to individual clients, or to any client that has registered for that kind of message. If necessary, receiving clients can send a response to the sender of a message they receive.

SAMP is in principle language agnostic and is based on abstract interfaces. That is, it specifies which functions the HUB and the clients must have in order to send and receive messages, but not the exact protocol that the HUB and client use to call those functions. The rules which describe how SAMP functions are mapped to the internally used protocol is described in a SAMP ‘Profile’. One standard profile based on XML-RPC is described in the official documentation, and this is what is used in Astro-WISE.

### SAMP datatypes¶

Because SAMP is language and even communication protocol agnostic, the number of datatypes that are supported is very limited. The only three data types are

1. string — a scalar value consisting of a sequence of ASCII-characters.
2. list — an ordered array of data items.
3. map — an unordered associative array with a string as key.

Other scalar types have to be mapped to strings, and there is a specification to represent integers, floats and booleans as strings. These data types can be nested to any level, e.g. it is possible to have a map with lists as values.

### SAMP messages¶

There is a small set of predefined message types, most (astronomically useful) message types are defined between application developers themselves. More or less agreed upon message types can be found on the SAMP wiki [5]. SAMP message types are abbreviated to MTypes.

An MType looks syntactically like a dot-separated string such as samp.hub.event.shutdown or table.load.votable. MTypes that start with samp. are administrative messages defined by the protocol, others are defined by application authors. Most messages have arguments such as the name of the file to be loaded. Messages can require the receiving end to send a return value, but so far no widely accepted MType does this. However, clients are supposed to give a general reply with success or failure of a requested operation even if no response is required.

• samp.hub.event.register
The hub broadcasts this message every time a client successfully registers.
Arguments:
• id (string) Public ID of newly registered client
• samp.hub.event.unregister
The hub broadcasts this message every time a client unregisters.
Arguments:
• id (string) public ID of unregistered client
• samp.hub.event.subscriptions
The hub broadcasts this message every time a client declares its subscriptions.
Arguments:
• id (string) public ID of subscribing client
• subscriptions (map) new subscriptions declared by client

### SAMP application messages¶

The relevant application defined messages for our purposes are

• table.load.votable
Loads a table in VOTable format.
Arguments:
• url (string) URL of the VOTable document to load
• table-id (string) (OPTIONAL) identifier which may be used to refer to the loaded table in subsequent messages
• name (string) (OPTIONAL) name which may be used to label the loaded table in the application GUI
• table.load.fits
Loads a table in FITS format.
Arguments:
• url (string) URL of the FITS file to load
• table-id (string) (OPTIONAL) identifier which may be used to refer to the loaded table in subsequent messages
• name (string) (OPTIONAL) name which may be used to label the loaded table in the application GUI
• table.highlight.row
Highlights a single row of an identified table by row index. The table to operate on is identified by one or both of the table-id or url arguments. At least one of these must be supplied; if both are given they should refer to the same thing. Exactly what highlighting means is left to the receiving application.
Arguments:
• table-id (string) identifier associated with a table by a previous message (e.g. table.load.*)
• url (string) URL of a table
• row (SAMP int) Row index (zero-based) of the row to highlight.
• table.select.rowList
Selects a list of rows of an identified table by row index. The table to operate on is identified by one or both of the table-id or url arguments. At least one of these must be supplied; if both are given they should refer to the same thing. Exactly what selection means is left to the receiving application.
Arguments:
• table-id (string) identifier associated with a table by a previous message (e.g. table.load.*)
• url (string) URL of a table
• row-list (list of SAMP int) list of row indices (zero-based) defining which table rows are to form the selection
• image.load.fits
Arguments:
• url (string) URL of the FITS image to load
• image-id (string) (OPTIONAL) Identifier which may be used to refer to the loaded image in subsequent messages
• name (string) (OPTIONAL) name which may be used to label the loaded image in the application GUI
• coord.pointAt.sky
Directs attention (e.g. by moving a cursor or shifting the field of view) to a given point on the celestial sphere.
Arguments:
• ra (SAMP float) right ascension in degrees
• dec (SAMP float) declination in degrees

## Query Driven Visualization Message Details¶

We designed new SAMP messages and data structures to enable query driven visualization through data pulling mechanisms. The object.* messages assume that the information system uses an object oriented model for science products such as catalogs.

### QDV SAMP: Message Types¶

The proposed messages are:

• catalog.derive: Create a catalog through data pulling. Arguments:
• catalog-id (string): Identifier of the catalog to select the sources from.
• query (string): Selection criterion for the sources.
• attributes (list of strings): Names of the attributes.
• catalog.pull: Perform the same action as catalog.derive and send the data over SAMP. Arguments:
• catalog-id (string): Identifier of the catalog to select the sources from.
• query (string): Selection criterion for the sources.
• attributes (list of strings): Names of the attributes.
• object.highlight: Highlight an object. Arguments:
• class (string): Class of the object.
• object-id (string): Identifier of the object.
• object.info: Returns a SAMP map with information about an object as described below. Arguments:
• class (string): Class of the object.
• object-id (string): Identifier of the object.
• object.change: Change a property of an object. Arguments:
• class (string): Class of the object.
• object-id (string): Identifier of the object.
• property-id (string): Identifier of a property of the object.
• value (string): New value of the property.
• object.action: Perform an action related to an object. Arguments:
• class (string): Class of the object.
• object-id (string): Identifier of the object.
• property-id (string): Identifier of a property of the object.
• action-id (string): Identifier of the action.

### QDV SAMP: Data Format¶

SAMP data structures are defined to send information about objects between applications. The structures are designed to be generic enough that they could be used for any information system.

Information about an object itself, e.g. the response to an object.info message, is communicated through a map with the following keys:

• class (string): The class of the object. A client that has knowledge about the used classes could handle known classes in a special way.
• id (string): Identifier this object, unique in combination with the class.
• status (string): Indication the processing status of this object (see below).
• properties (list of maps): Properties of this object (see below).
• actions (list of maps): Actions that can be performed on this object (see below).
• readonly (boolean): Flag to indicate that the object cannot be modified.

Properties of an object, for example process parameters, are described with a map with the following keys:

• name (string): Name of the property, as used in the object.
• class (string): The class that the value of the property should have, or a primitive such as ‘int’.
• description (string): A human readable description of the property.
• value (string): The used value for the property. This is the id of the object if the property references to a persistent class.
• options (list of maps): Possible values for the property, if applicable (see below).
• actions (list of maps): Actions that can be performed on the property (see below).
• readonly (boolean): Flag to indicate that the property cannot be modified.

An action that can be performed on an object or property is defined by a map with the following keys:

• id (string): A unique identifier for this action.
• name (string): A human presentable name for this action.

### QDV Samp: Object Status¶

The status value of an object refers to the processing status of the object. It can have the following values:

• ok: The object has been processed, or can be processed while retrieving the result.
• automatic: The object has to be processed before the can be retrieved. This can be done without user interaction.
• new: This is a non persistent object, which can be processed without user interaction.
• depends: This is a new object, which can be processed only after human intervention. For example to set a process parameter that has no proper default.
• not: As it is, this object cannot be processed, e.g. because a dependency cannot be fulfilled. The scientist might be able to solve the problem, but whether this is the case is not clear to the information system.
• unknown: The status is unknown.

The awe-prompt SAMP client currently does not support the status property, it returns unknown on all objects.

### QDV Actions¶

The actions value of the dictionaries refer to actions that can be performed through the object.action message.