IVOA Universal Worker Service

What is UWS?

IVOA Universal Worker Services are standardised web services that allow the execution of predefined asynchroneous jobs (for specifications see: IVOA UWS standard). Some services might probide cone search facilities, others might provide more sophisticated services. With Daiquiri you have the possibility to run queries on data through a UWS service. You can even implement your own service for other purposes by soley deriving from the UwsAbstract.php class... But let's not get into this yet.

The UWS is specified as a REST service, so you can send simple GET, POST, PUT, and DELETE requests to the UWS service using "httpie" or any other tool you might want to use. Unfortunately "curl" won't do this, since it has some major issues regarding its HTTP 1.1 protocol implementation.

In order to work from the command line with UWS services, please install "httpie" from https://github.com/jkbr/httpie! It makes life easier and more worthwhile...

Since REST requires a web service to be "state less", the way you have to work might seem odd at first. But your workflow can be easily scripted away. We might even provide a Python library in the future for easy job submission. But for the moment you need to form the request by hand - an easy task we will describe below.

A Brief Description

The Daiquiri UWS service resides at {base_daiquiri_URL}/uws/{name_of_service}. For instance, let's assume that {base_daiquiri_URL} is escience.aip.de/daiquiri. The {name_of_service} specifies the UWS service you want to use. At the moment, the only implemented UWS service is query, but there might be different services implemented. So for this tutorial we will use:

http://escience.aip.de/daiquiri/uws/query

Now that we have located our UWS service, we can start using it. Before we do that though, let's remind ourselves what the various REST / HTTP request methods do:

Requests are authenticated using "Basic" HTTP authentication and are easily supported by tools like httpie. A basic request to the Daiquiri UWS service will thus look like (using httpie):

FOR GET:    http --auth username:password GET http://example.com/uws
FOR POST:   http --auth username:password --form --follow POST http://example.com/uws param1=foo param2=bar
FOR PUT:    echo "YOUR_PUTTABLE_CONTENT" | http --auth username:password --follow PUT http://example.com/uws/resource
FOR DELETE: http --auth username:password --follow DELETE http://example.com/uws/resource

Obtaining a List of Jobs

You can obtain a list of your UWS jobs by sending a GET request to the base URI of the service:

http --auth username:password GET http://escience.aip.de/daiquiri/uws/query

You will then obtain the result as IVOA compliant XML file as described in the IVOA standard:

<?xml version="1.0" encoding="UTF-8"?>
<uws:jobs xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink">
  <uws:jobref id="2013-05-30T14:38:37:9708" xlink:href="http://escience.aip.de/daiquiri/uws/query/350698875526375" xlink:type="simple">
    <uws:phase>COMPLETED</uws:phase>
  </uws:jobref>
  <uws:jobref id="2013-05-30T11:17:56:2845" xlink:href="http://escience.aip.de/daiquiri/uws/query/350695801893750" xlink:type="simple">
    <uws:phase>COMPLETED</uws:phase>
  </uws:jobref>
  <uws:jobref id="2013-05-28T10:19:17:7360" xlink:href="http://escience.aip.de/daiquiri/uws/query/350650664387723481" xlink:type="simple">
    <uws:phase>ERROR</uws:phase>
  </uws:jobref>
  </uws:jobref>
  <uws:jobref id="2013-04-25T14:09:25:6032" xlink:href="http://escience.aip.de/daiquiri/uws/query/349924291999050698" xlink:type="simple">
    <uws:phase>COMPLETED</uws:phase>
  </uws:jobref>
</uws:jobs>

The XML is structured in such a way, that all jobs are enclosed in the root node "uws:jobs" which then contains a list of "uws:jobref"s. "uws:jobref" are as the name suggests references to jobs in the UWS service. They contain a link to the job and specify the job status as a "uws:phase" tag. The following UWS job phases exist (as given in the standard):

Creating a New Job

Creating a new job is done by POSTing an empty string to the base URI of the service:

http --auth username:password --form --follow POST http://escience.aip.de/daiquiri/uws/query

This results in an empty job like this:

<?xml version="1.0" encoding="UTF-8"?>
<uws:job xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <uws:jobId>350719894773373</uws:jobId>
  <uws:runId></uws:runId>
  <uws:ownerId>admin</uws:ownerId>
  <uws:phase>PENDING</uws:phase>
  <uws:quote xsi:nil="true"/>
  <uws:startTime xsi:nil="true"/>
  <uws:endTime xsi:nil="true"/>
  <uws:executionDuration>0</uws:executionDuration>
  <uws:destruction>2999-12-31T00:00:00+01:00</uws:destruction>
  <uws:parameters/>
  <uws:results/>
</uws:job>

which you can also retrieve with the jobId by sending the following GET request to {base_daiquiri_URL}/uws/{name_of_service}/{jobId}

http --auth username:password GET http://escience.aip.de/daiquiri/uws/query/350719894773373

The job has various parameters that are either set by the server and are immutable (like jobId, ownerId, quote, startTime, endTime, results) and ones that can be changed (runId, phase, executionDuration, destruction). These are the parameters required by UWS to work. You can also specify parameters that are needed by the service you are running. These you will find in the tag "uws:parameters".

NOTE: The Daiquiri implementation of UWS does not support destruction times yet, so these are always set to a ridiculously high value. Further, the quoting system is not implemented for normal query processes, since this is a rather hard task. We set the quote to unknown (see the standard for further information).

Setting UWS Parameters

UWS parameters that you can set are executionDuration and destruction (however Daiquiri just ignores setting destruction for the moment). We will describe how to set service related parameters in the next section.

You can set the runId and executionDuration by sending a POST request to {base_daiquiri_URL}/uws/{name_of_service}/{jobId}/destruction or {base_daiquiri_URL}/uws/{name_of_service}/{jobId}/executionDuration. As a side note, you can issue a GET request to the respective URIs to obtain the value of the parameter (e.g. {base_daiquiri_URL}/uws/{name_of_service}/{jobId}/quote for a quote). Such a POST request would look like this:

http --auth username:password --form --follow POST http://escience.aip.de/daiquiri/uws/query/350719894773373/executionduration executionduration=20

Note how UWS returns the complete job object and how the executionDuration changed from 0 to 20.

NOTE: If the executionDuration is set to 0, this is according to the UWS standard equal to unlimited execution. Daiquiri does not support this necessarily and if a queue is used for the SQL queries, the default queue will be used!

NOTE: UWS requires UWS parameters to be set using POST and not PUT. This is somewhat contrary to the REST pattern.

UWS service related parameters are parameters needed to execute the job and serve as input parameters (in contrast to the UWS parameters that govern the queueing of the job). A list of set service parameters can be queried by sending a GET request to {base_daiquiri_URL}/uws/{name_of_service}/{jobId}/parameters.

The Daiquiri UWS query service currently supports the following parameters:

When setting a parameter and it does not yet exists, it will be allocated in the job description. However you cannot delete the parameter anymore, this is not supported by the UWS standard. You can set parameters using three different methods (why one if you can do it differently...):

Setting Parameters on Job Creation

You can also any parameter while creating the job when you POST all the parameter value pairs:

http --auth username:password --form --follow POST http://escience.aip.de/daiquiri/uws/query executionduration=20 query="select * from daiquiri_test.FOO" table="my_great_result"

This will result in a new job with the following form:

<?xml version="1.0" encoding="UTF-8"?>
<uws:job xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <uws:jobId>350722572652862</uws:jobId>
  <uws:runId></uws:runId>
  <uws:ownerId>admin</uws:ownerId>
  <uws:phase>PENDING</uws:phase>
  <uws:quote xsi:nil="true"/>
  <uws:startTime xsi:nil="true"/>
  <uws:endTime xsi:nil="true"/>
  <uws:executionDuration>20</uws:executionDuration>
  <uws:destruction>2999-12-31T00:00:00+01:00</uws:destruction>
  <uws:parameters>
    <uws:parameter id="query">select * from daiquiri_test.FOO</uws:parameter>
    <uws:parameter id="table">my_great_result</uws:parameter>
  </uws:parameters>
  <uws:results/>
</uws:job>

Submitting a Job

If the job has been set up to your liking, you can then submit the job to the UWS queue. In the case of the query, this might either be that the job is executed immediately or that it will be queued. This depends on the way Daiquiri is used on the server. When you submit a job, the correct queue is determined using the exeuctionDuration that you have defined. It will then be set accordingly.

To submit a job you send a POST request to the job URI with the parameter "phase=run":

http --auth username:password --form --follow POST http://escience.aip.de/daiquiri/uws/query/350719894773373 phase=run

If the job is accepted by the queue, it will be run and you can query its state by GET requesting the "phase" parameter of the job.

http --auth username:password GET http://escience.aip.de/daiquiri/uws/query/350719894773373/phase

If in the case of the query, any SQL query validation errors will array, the job's state will be set to ERROR and you can change the parameters (this is a breach of the UWS standard but we thought this might be a convenient solution). If the error has been raised by the queue and database, you are out of luck and need to create a new job for fixing the problem. You can query any error by GET requesting the "error" parameter of the job.

http --auth username:password GET http://escience.aip.de/daiquiri/uws/query/350719894773373/error

Aborting a Job

If you need to abort a job, you can send a POST request to the job URI with the parameter "phase=abort". The UWS service will then try to abort the job.

http --auth username:password --form --follow POST http://escience.aip.de/daiquiri/uws/query/350719894773373 phase=abort

Retrieving Results

To retrieve the results of the job, you send a GET request to the job URI. You will then either obtain the results XML encoded or you get a list of references with URLs where you can pickup the results:

http --auth username:password GET http://escience.aip.de/daiquiri/uws/query/350719894773373/results

In the case of the Daiquiri query facility you will get something like:

<?xml version="1.0" encoding="UTF-8"?>
<uws:results xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink">
  <uws:result id="2013-05-30T14:38:37:9708" xlink:href="http://escience.aip.de/daiquiri/query/index/stream/table/2013-05-30T14%3A38%3A37%3A9708/format/csv" xlink:type="simple"/>
</uws:results>

The URL given in the "xlink:href" parameter is the one you need to call to obtain the results (in the example above this is "http://escience.aip.de/daiquiri/query/index/stream/table/2013-05-30T14%3A38%3A37%3A9708/format/csv").

Deleting Jobs

If for any reason you want to delete a job, you send a DELETE request (or a POST with "action=DELETE") to the job URI. You can only issue DELETE requests to jobs and no other resource.

http --auth username:password --follow DELETE http://escience.aip.de/daiquiri/uws/query/350719894773373