REST API Tutorial 12: GUI control calls: immediate extract

Last update	Dec 2023
Environment	Any
Language	Any HTTP is supported
Compilers	None
Prerequisites	DSS login, internet access
Source code	Below

Tutorial purpose

This tutorial goes through the basics of programmatic GUI control, following the entire workflow for a scheduled data extraction.

Instead of using an On Demand extraction, which is a simplified query, it performs a whole set of actions that could also be done manually in the web GUI. This means that the result of the actions described in this tutorial can also be seen in the web GUI, which gives you a good testing tool.

For an explanation on these two approaches, look here.

In this tutorial a Time and Sales (Tick) data extraction is performed. Other extractions will work in a similar way.

Basic operations: explanations
Retrieve the available field list from the server
Create an instrument list
Append instruments to an instrument list
Create a report template
Schedule an immediate extraction
Check the extraction status
Retrieve the extraction report
Retrieve the data from the server
Clean up
Error handling

Basic operations: explanations

Before explaining the code, let us briefly explain what we are trying to achieve.

To retrieve data, 3 basic elements are required:

An instrument list. This is the list of financial instrument identifiers for which data will be requested. Using the API, an instrument list can be created, and instruments added to it.
A report template. It defines the format of the requested data, and the list of fields. Depending on the requested data (pricing data, reference data, news, estimates, corporate actions, etc.), different report options and data fields are available. Using the API, a report template can be created, and its data fields selected.
An extraction schedule. It defines when the data will be retrieved, and triggers the actual data extraction. Its frequency could be once-off or recurring. Its timing could be immediate, at a fixed time, or triggered when data is available.

They are all stored on the DSS server, and you can have several of each.

They can be created, displayed, modified and deleted manually, using the web GUI, and that might be the best solution if they are invariable in time and only need to be created once. But regular changes to some or all of these 3 elements might be required, and the purpose of the API is to give us the tools to automate that. The most frequent use case is the need to change the instrument list, which could be on a daily basis.

The purpose of this tutorial is to demonstrate how to do these things programmatically, using the LSEG Tick History REST API.

If your use case only requires regular changes to an instrument list, you could decide to manually create an empty instrument list, a report template and an extraction schedule, and then create a program that automatically changes the contents of the instrument list. You could also do it all programmatically, which is easier.

Retrieve the available field list from the server

Get available field list - HTTP request

This step is exactly the same as what we did at the start of Tutorial 4, but for the sake of completeness we repeat it here. All the other steps are new.

This step is optional. If you do not know what content field names are available, you can request a list of those available

URL:

The available field set depends on the type of data you want to request. The data type must therefore be specified in the request, which is done by setting a report template type in the URL. As we want tick data we set a value of TickHistoryTimeAndSales.

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/GetValidContentFieldTypes(ReportTemplateType=DataScope.Select.Api.Extractions.ReportTemplates.ReportTemplateTypes'TickHistoryTimeAndSales')

Method: GET

Headers:

Note: for all requests we need a user token. The token was retrieved in Tutorial 1.

    	
            Prefer: respond-async
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Get available field list - HTTP response

If the token is valid, this is the response we get:

Status: 200 OK

Relevant headers:

    	
            Content-Type: application/json; charset=utf-8

Body:

There are more than 300 values in the response. Here is the beginning of the response:

    	
            {
  "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ContentFieldTypes",
  "value": [
    {
      "Code": "THT.Auction - Exchange Time",
      "Name": "Auction - Exchange Time",
      "Description": "Exchange supplied exchange time (Local or GMT depending on the exchange).",
      "FormatType": "Text",
      "FieldGroup": "Auction"
    },
    {
      "Code": "THT.Auction - Price",
      "Name": "Auction - Price",
      "Description": "Auction Price.",
      "FormatType": "Number",
      "FieldGroup": "Auction"
    },
    {
      "Code": "THT.Auction - Qualifiers",
      "Name": "Auction - Qualifiers",
      "Description": "Trade qualifiers or market condition indicator; See Qualifiers for more details.",
      "FormatType": "Text",
      "FieldGroup": "Auction"
    },
    {
      "Code": "THT.Auction - Sequence Number",
      "Name": "Auction - Sequence Number",
      "Description": "An exchange derived sequence number associated with the auction (applicable to US markets).",
      "FormatType": "Number",
      "FieldGroup": "Auction"
    },
    {
      "Code": "THT.Auction - Volume",
      "Name": "Auction - Volume",
      "Description": "Auction Volume.",
      "FormatType": "Number",
      "FieldGroup": "Auction"
    },

This goes on with all the other available fields. Here is the last part:

    	
                {
      "Code": "THT.Trade - Volume",
      "Name": "Trade - Volume",
      "Description": "Last traded volume",
      "FormatType": "Number",
      "FieldGroup": "Trade"
    },
    {
      "Code": "THT.Trade - Weak Market",
      "Name": "Trade - Weak Market",
      "Description": "Market weakness",
      "FormatType": "Number",
      "FieldGroup": "Trade"
    },
    {
      "Code": "THT.Trade - Yield",
      "Name": "Trade - Yield",
      "Description": "An update to indicate Dividend Yield as adjusted by the last trade or closing price.",
      "FormatType": "Number",
      "FieldGroup": "Trade"
    }
  ]
}

The result contains the field code, name, a description, field type (number, text, date) and group. Use this to choose the field names you want. In the next step we will make a request for data, using some data fields we chose.

Create an instrument list

Create an instrument list - HTTP request

In this step we create an empty instrument list. We will populate it in the next step, by appending instrument identifiers to the list.

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/InstrumentLists

Method: POST

Headers:

    	
            Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Body:

The body of the request must mention:

The data type, which is an InstrumentList.
The name of the instrument list. This must be unique, an error will be generated if the name already exists.

    	
            {
  "@odata.type": "#DataScope.Select.Api.Extractions.SubjectLists.InstrumentList",
  "Name": "myInstrumentListName"
}

Create an instrument list - HTTP response

If the token is valid, and an instrument list with the same name does not yet exist, this is the response we get:

Status: 201 Created

Relevant headers:

    	
            Content-Type: application/json; charset=utf-8
Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/InstrumentLists('')

Body:

    	
            {
    "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#InstrumentLists/$entity",
    "ListId": "0x07861e8bf35c779d",
    "Name": "myInstrumentListName",
    "Count": 0,
    "Created": "2021-04-20T06:29:10.127Z",
    "Modified": "2021-04-20T06:29:10.127Z"
}

Your application must cache the value of ListId returned in the body content, it will be required in the next steps.

In case of error, see the Error handling section below.

Create an instrument list - check in the DSS web GUI

Let us display the result of this operation in the web GUI: click on DATASCOPE SELECT top left, then on Instrument Lists:

At this stage, the instrument list is created, but when we click on it we see it is empty:

If there are many and you can’t find it, sort the entries by clicking on the Creation Date column header.

Append instruments to an instrument list

Append instruments to an instrument list - HTTP request

Intrument identifiers are added to the instrument list by appending them. This can even be done several times.

The request path refers to the instrument list Id (0x07861e8bf35c779d) which was returned when we created the instrument list in the previous step.

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/InstrumentLists('0x07861e8bf35c779d')/DataScope.Select.Api.Extractions.InstrumentListAppendIdentifiers

Method: POST

Headers:

    	
            Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Body:

The body of the request must mention:

The list of instrument identifiers, each one with its type (Cusip, Isin, Ric, etc., more than 30 are available). The user defined identifier is optional; it is just an additional tag that will be stored in the instrument list on the server. It could contain a customer internal identifier, asset class, industry sector, country, currency, etc. Instrument identifiers can be filtered or sorted by this tag in the web GUI.
In the example below we define one instrument using a CUSIP code (Committee on Uniform Securities Identification Procedures, an identification system for north American and Canadian stocks), and another one using a RIC. More information on them can be found by browsing in the API Reference Tree or searching in the help in the web GUI:

We also include, for training purposes, an invalid instrument, to see the effects of instrument validation.
What to do if there is a duplicate instrument identifier.

    	
            {
  "Identifiers": [
    {
      "Identifier": "CARR.PA",
      "IdentifierType": "Ric",
      "UserDefinedIdentifier": "EQUITYTEST"
    },
    {
      "Identifier": "438516AC0",
      "IdentifierType": "Cusip",
      "UserDefinedIdentifier": "BONDTEST"
    },
    {
      "Identifier": "JUNK.JUNK",
      "IdentifierType": "Ric",
      "UserDefinedIdentifier": "INVALID"
    }
  ],
  "KeepDuplicates": false
}

Append instruments to an instrument list - HTTP response

If the token is valid, this is the response we get:

Status: 200 OK

Relevant headers:

    	
            Content-Type: application/json; charset=utf-8

Body:

    	
            {
  "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#DataScope.Select.Api.Extractions.SubjectLists.InstrumentsAppendIdentifiersResult",
  "ValidationResult": {
    "ValidInstrumentCount": 2,
    "OpenAccessSegments": [],
    "StandardSegments": [
      {
        "Code": "E",
        "Description": "Equity",
        "Count": 1
      },
      {
        "Code": "G",
        "Description": "GORP",
        "Count": 1
      }
    ],
    "ValidationDuplicates": [],
    "Messages": [
      {
        "Severity": "Info",
        "Message": "RIC, JUNK.JUNK (not found)"
      }
    ]
  },
  "AppendResult": {
    "AppendedInstrumentCount": 2,
    "AppendDuplicates": []
  }
}

The body contains several sections:

Instrument identifiers validation results, including the number of valid instruments, and some high level information on the segmentation by asset class of the instrument identifiers.
If there are issues, a messages section will list all invalid instruments.
Instrument identifiers append results, including the number of appended instruments, and details on eventual duplicates,

Append instruments to an instrument list - check in the DSS web GUI

The result of this operation can be seen in the web GUI. Now we find our 2 instruments in the instrument list:

Create a report template

Create a report template - HTTP request

The report template type must be set in the path of the request. In this case it is a Time and Sales (tick) report template.

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/TickHistoryTimeAndSalesReportTemplates

Method: POST

Headers:

    	
            Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Body:

The body of the request must mention several parameters. The main ones are:

The data type, which in this case is an TickHistoryTimeAndSalesReportTemplate.
Some parameters to customise the output file headers and trailers.
The name of the report template. This must be unique, an error will be generated if the name already exists.
The list of data content fields we want in the result. These were chosen among all the available ones, see Retrieve the available field list from the server above.
For more information on available report templates and corresponding fields, see here.

    	
            {
  "@odata.type": "#DataScope.Select.Api.Extractions.ReportTemplates.TickHistoryTimeAndSalesReportTemplate",
  "ShowColumnHeaders": false,
  "Name": "myTandSTemplateName",
  "Headers": [],
  "Trailers": [],
  "ContentFields": [
    { "FieldName": "Quote - Ask Price" },
    { "FieldName": "Quote - Ask Size" },
    { "FieldName": "Quote - Bid Price" },
    { "FieldName": "Quote - Bid Size" },
    { "FieldName": "Quote - Exchange Time" }
  ],
    "Condition": {
      "MessageTimeStampIn": "GmtUtc",
      "ApplyCorrectionsAndCancellations": false,
      "ReportDateRangeType": "Range",
      "QueryStartDate": "2016-09-29T15:15:00.000Z",
      "QueryEndDate": "2016-09-29T15:45:00.000Z"
    }
}

Create a report template - HTTP response

If the token is valid, and a report template with the same name does not yet exist, this is the response we get:

Status: 201 Created

Relevant headers:

    	
            Content-Type: application/json; charset=utf-8
Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/TickHistoryTimeAndSalesReportTemplates('')

Body:

    	
            {
    "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#TickHistoryTimeAndSalesReportTemplates/$entity",
    "ReportTemplateId": "0x07861c85fc0c7796",
    "ShowColumnHeaders": true,
    "CompressionType": "GZip",
    "CreateDate": "2021-04-20T06:41:40.290Z",
    "LastChangedDate": "2021-04-20T06:41:40.290Z",
    "Name": "myTandSTemplateName",
    "OutputFormat": "CommaSeparatedValues",
    "ReportFieldCount": 5,
    "Delimiter": "None",
    "DeliveryType": "None",
    "TemplateTypeCode": "THT",
    "Headers": [],
    "Trailers": [],
    "ContentFields": [
        {
            "FieldName": "Quote - Ask Price",
            "Justification": "Center",
            "WidthStyle": "VariableWidth",
            "Format": {
                "@odata.type": "#DataScope.Select.Api.Extractions.ReportTemplates.ContentFieldNumberFormat",
                "DecimalPlaces": 9,
                "DecimalSeparator": "Comma",
                "IntegerPlaces": 18,
                "UseLeadingZero": false,
                "NegativeSignPosition": "Before",
                "ThousandSeparator": "Space",
                "UseThousandSeparator": true,
                "UseTrailingZero": false
            }
        },
        {
            "FieldName": "Quote - Ask Size",
            "Justification": "Center",
            "WidthStyle": "VariableWidth",
            "Format": {
                "@odata.type": "#DataScope.Select.Api.Extractions.ReportTemplates.ContentFieldNumberFormat",
                "DecimalPlaces": 9,
                "DecimalSeparator": "Comma",
                "IntegerPlaces": 18,
                "UseLeadingZero": false,
                "NegativeSignPosition": "Before",
                "ThousandSeparator": "Space",
                "UseThousandSeparator": true,
                "UseTrailingZero": false
            }
        },
        {
            "FieldName": "Quote - Bid Price",
            "Justification": "Center",
            "WidthStyle": "VariableWidth",
            "Format": {
                "@odata.type": "#DataScope.Select.Api.Extractions.ReportTemplates.ContentFieldNumberFormat",
                "DecimalPlaces": 9,
                "DecimalSeparator": "Comma",
                "IntegerPlaces": 18,
                "UseLeadingZero": false,
                "NegativeSignPosition": "Before",
                "ThousandSeparator": "Space",
                "UseThousandSeparator": true,
                "UseTrailingZero": false
            }
        },
        {
            "FieldName": "Quote - Bid Size",
            "Justification": "Center",
            "WidthStyle": "VariableWidth",
            "Format": {
                "@odata.type": "#DataScope.Select.Api.Extractions.ReportTemplates.ContentFieldNumberFormat",
                "DecimalPlaces": 9,
                "DecimalSeparator": "Comma",
                "IntegerPlaces": 18,
                "UseLeadingZero": false,
                "NegativeSignPosition": "Before",
                "ThousandSeparator": "Space",
                "UseThousandSeparator": true,
                "UseTrailingZero": false
            }
        },
        {
            "FieldName": "Quote - Exchange Time",
            "Justification": "Center",
            "WidthStyle": "VariableWidth",
            "Format": {
                "@odata.type": "#DataScope.Select.Api.Extractions.ReportTemplates.ContentFieldTextFormat",
                "Capitalization": "None"
            }
        }
    ],
    "Condition": {
        "SortBy": "SingleByRic",
        "MessageTimeStampIn": "GmtUtc",
        "TimeRangeMode": "",
        "ApplyCorrectionsAndCancellations": false,
        "ReportDateRangeType": "Range",
        "QueryStartDate": "2016-09-29T15:15:00.000Z",
        "QueryEndDate": "2016-09-29T15:45:00.000Z",
        "Preview": "None",
        "ExtractBy": "Ric",
        "DisplaySourceRIC": false
    }
}

Your application must cache the value of ReportTemplateId returned in the body content, it will be required in the next steps.

In case of error, see the Error handling section below.

Create a report template - check in the web GUI

Let us display the result of this operation in the web GUI: click on DATASCOPE SELECT top left, then on Report Templates:

We find our report template with 5 fields:

If there are many and you can’t find it, sort the entries by clicking on the Creation Date column header.

Schedule an immediate extraction

Schedule an immediate extraction - HTTP request

An extraction schedule defines when the data will be retrieved, and triggers the actual extraction. It must refer to an instrument list (for which the extraction will be done), and a report template (defining what data needs to be extracted).

An extraction schedule can run once or recurring. Its timing can be immediate, at a fixed time, or triggered when data is available.

It is possible to define several schedules.

In this tutorial we define an extraction schedule for our previously created instrument list and report template, that will be run immediately, only once.

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/Schedules

Method: POST

Headers:

    	
            Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Body:

The body of the request must mention several parameters. The main ones are:

The name of the extraction schedule. This must be unique, an error will be generated if the name already exists.
The recurrence. In this example we define it to be single and immediate.
The trigger. In this example we define an immediate trigger.
The instrument list id. Its value is that which was returned when we created the instrument list.
The report template id. Its value is that which was returned when we created the report template.

    	
            {
  "Name": "myImmediateSchedule",
  "Recurrence": {
    "@odata.type": "#DataScope.Select.Api.Extractions.Schedules.SingleRecurrence",
    "IsImmediate": true
  },
  "Trigger": {
    "@odata.type": "#DataScope.Select.Api.Extractions.Schedules.ImmediateTrigger"
  },
  "ListId": "0x07861e8bf35c779d",
  "ReportTemplateId": "0x07861c85fc0c7796"
}

Schedule an immediate extraction - HTTP response

If the token is valid, and a schedule with the same name does not yet exist, this is the response we get:

Status: 201 Created

Relevant headers:

    	
            Content-Type: application/json; charset=utf-8
Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/Schedules('')

Body:

    	
            {
    "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#Schedules/$entity",
    "ScheduleId": "0x07861d8f7ebc7797",
    "Name": "myImmediateSchedule",
    "TimeZone": "UTC",
    "Recurrence": {
        "@odata.type": "#DataScope.Select.Api.Extractions.Schedules.SingleRecurrence",
        "ExtractionDateTime": "0001-01-01T00:00:00Z",
        "IsImmediate": true
    },
    "Trigger": {
        "@odata.type": "#DataScope.Select.Api.Extractions.Schedules.ImmediateTrigger",
        "LimitReportToTodaysData": false
    },
    "UserId": 9008895,
    "CreateDate": "2021-04-20T06:59:11.609Z",
    "LastChangeDate": "2021-04-20T06:59:11.609Z",
    "ListId": "0x07861e8bf35c779d",
    "ReportTemplateId": "0x07861c85fc0c7796"
}

Most of the returned content is a recap of the request, but with added details. As the extraction is scheduled to be immediate, the ExtractionDateTime and TimeZone will not be used.

Your application must cache the value of ScheduleId returned in the body content, it will be required later.

In case of error, see the Error handling section below.

Schedule an immediate extraction - check in the DSS web GUI

Let us display the result of this operation in the web GUI: click on DATASCOPE SELECT top left, then on Schedules:

We find our schedule, as defined. Immediately after sending the request, its status shows it has been Submitted and is Processing:

Once the scheduled data extraction has completed, its last status changes to Completed:

If there are many and you can’t find it, sort the entries by clicking on the Creation Date column header.

Check the extraction status

Extraction status life cycle

The extraction will first be queued, its status will be pending. Once triggered, it will be processed. Once all the data is extracted, it will be complete.

The following chart illustrates the extraction events as they progress in time from left to right, with the related values of the extraction Status and DetailedStatus:

In the previous step we showed how to check the schedule and extraction status in the web GUI, which is fine for testing, but it is also possible to do this programmatically.

Check the extraction status - HTTP request

The schedule id we received in the response to our schedule creation request is used as a parameter set in the path of the request.

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/Schedules('0x07861d8f7ebc7797')/LastExtraction

Method: GET

Headers:

    	
            Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Check the extraction status - HTTP response

If the schedule has not yet completed, an HTTP status 204 is returned (the body of such a response is empty). We must wait, then check the extraction status again, until receiving an HTTP status 200:

Status: 200 OK

Relevant headers:

    	
            Content-Type: application/json; charset=utf-8

Body:

    	
            {
    "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ReportExtractions/$entity",
    "ReportExtractionId": "2000000250046155",
    "ScheduleId": "0x07861d8f7ebc7797",
    "Status": "Completed",
    "DetailedStatus": "Done",
    "ExtractionDateUtc": "2021-04-20T06:59:12.140Z",
    "ScheduleName": "myImmediateSchedule",
    "IsTriggered": false
}

Report Extraction - HTTP request

The ReportExtractionId we received in the response is used as a parameter set in the path of the request to list extraction files.

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ReportExtractions('2000000250046155')/Files

Method: GET

Headers:

    	
            Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Retrieve the list of extraction files - HTTP response

If the token is valid, this is the response we get:

Status: 200 OK

Relevant headers:

    	
            Content-Type: application/json; charset=utf-8

Body:

    	
            {
    "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ExtractedFiles",
    "value": [
        {
            "ExtractedFileId": "VjF8MHgwNzg2MGQ0MDJiMWM3NzNifA",
            "ReportExtractionId": "2000000250046155",
            "ScheduleId": "0x07861d8f7ebc7797",
            "FileType": "Full",
            "ExtractedFileName": "9008895.myImmediateSchedule.20210420.065912.2000000250046155.x02q13.csv.gz",
            "LastWriteTimeUtc": "2021-04-20T07:01:04.000Z",
            "ContentsExists": true,
            "Size": 74974
        },
        {
            "ExtractedFileId": "VjF8MHgwNzg2MjAwMzRkZWM3NzlmfA",
            "ReportExtractionId": "2000000250046155",
            "ScheduleId": "0x07861d8f7ebc7797",
            "FileType": "Note",
            "ExtractedFileName": "9008895.myImmediateSchedule.20210420.065912.2000000250046155.x02q13.csv.gz.notes.txt",
            "LastWriteTimeUtc": "2021-04-20T07:01:04.480Z",
            "ContentsExists": true,
            "Size": 1889,
            "ReceivedDateUtc": "2021-04-20T07:01:04.480Z"
        }
    ]
}

Note. in the result set, the order of the files can vary.

We see 2 files have been generated on the DSS server:

The Full extracted data itself.
The extraction Note file, which contains details of the extraction.

Each file has an ExtractedFileId. Your application must cache the values of these file ids, to be able to access the files contents.

Retrieve the data from the server

Once the status is Completed, and we have retrieved the ExtractedFileId, we can retrieve the data.

Retrieve the data from the server - HTTP request

The extracted file id we received for the Full data file in the response to our extraction report request is used as a parameter set in the path of the request.

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractedFiles('VjF8MHgwNzg2MGQ0MDJiMWM3NzNifA')/$value

Method: GET

Headers:

    	
            Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Retrieve the data from the server - HTTP response

If the token is valid, this is the response we get:

Status: 200 OK

Relevant headers:

    	
            Content-Type: text/plain

Note: contrary to most responses, this time the content is not in JSON format !

Body:

Here is the beginning of the data (there is much more):

    	
            #RIC,Domain,Date-Time,GMT Offset,Type,Bid Price,Bid Size,Ask Price,Ask Size,Exch Time
CARR.PA,Market Price,2016-09-29T13:15:00.097355899Z,+2,Quote,23.09,3085,23.095,100,13:15:00.048000000
CARR.PA,Market Price,2016-09-29T13:15:00.097355899Z,+2,Quote,23.09,3120,23.095,100,13:15:00.049000000
CARR.PA,Market Price,2016-09-29T13:15:01.033433922Z,+2,Quote,23.09,3370,23.095,100,13:15:00.977000000
CARR.PA,Market Price,2016-09-29T13:15:01.033433922Z,+2,Quote,23.09,3370,23.1,713,13:15:00.977000000
CARR.PA,Market Price,2016-09-29T13:15:01.033433922Z,+2,Quote,23.095,1700,23.1,713,13:15:00.977000000
CARR.PA,Market Price,2016-09-29T13:15:01.033433922Z,+2,Quote,23.095,1700,23.1,300,13:15:00.978000000
CARR.PA,Market Price,2016-09-29T13:15:01.033433922Z,+2,Quote,23.095,1900,23.1,300,13:15:00.978000000
CARR.PA,Market Price,2016-09-29T13:15:01.033433922Z,+2,Quote,23.095,2170,23.1,300,13:15:00.978000000
CARR.PA,Market Price,2016-09-29T13:15:01.037513480Z,+2,Quote,23.095,2346,23.1,300,13:15:00.979000000
...

The content of the file is the data, in CSV format. After the default fields (RIC, data type, time stamp, record type, we find the 5 requested fields (bid and ask price and size, and the exchange time). Depending on the instruments, chosen fields and time of request, it can happen that no data is available for some fields.

Retrieve the extraction notes from the server - HTTP request

The example above illustrated how to retrieve the contents of the data file. Retrieving the contents of the other files can be done using their respective ExtractedFileId values. The extracted file id we received for the Note file in the response to our extraction report request is used as a parameter set in the path of the request.

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractedFiles('VjF8MHgwNzg2MjAwMzRkZWM3NzlmfA')/$value

Method: GET

Headers:

    	
            Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Retrieve the extraction notes from the DSS server - HTTP response

If the token is valid, this is the response we get:

Status: 200 OK

Relevant headers:

    	
            Content-Type: text/plain

Body:

    	
            Extraction Services Version 14.5.42294 (737b0965c07f), Built Apr  8 2021 13:43:46
User ID: 9008895
Extraction ID: 2000000250046155
Schedule: myImmediateSchedule (ID = 0x07861d8f7ebc7797)
Input List (2 items): myInstrumentListName (ID = 0x07861e8bf35c779d) Created: 04/20/2021 06:29:10 Last Modified: 04/20/2021 06:38:37
Report Template (5 fields): myTandSTemplateName (ID = 0x07861c85fc0c7796) Created: 04/20/2021 06:41:40 Last Modified: 04/20/2021 06:41:40
Schedule dispatched via message queue (0x07861d8f7edc7797), Data source identifier (F382DA52FA7A47BE84965537FADFA1D1)
Schedule Time: 04/20/2021 06:59:12
Processing started at 04/20/2021 06:59:21
Processing completed successfully at 04/20/2021 07:01:04
Extraction finished at 04/20/2021 07:01:04 UTC, with servers: tm02n01
Instrument <RIC,CARR.PA> expanded to 1 RIC: CARR.PA.
Instrument <CSP,438516AC0> expanded to 2 RIC instances: TESTFINN=TEST to US438516AC05=JPNY.
Total instruments after instrument expansion = 3
Quota Message: INFO: Tick History Cash Quota Count Before Extraction: 3190; Instruments Approved for Extraction: 0; Tick History Cash Quota Count After Extraction: 3190, 638% of Limit; Tick History Cash Quota Limit: 500
Quota Message: ERROR: The RIC 'CARR.PA' in the request would exceed your quota limits. Adjust your input list to continue.
Quota Message: WARNING: Tick History Cash Quota has been reached or exceeded
Quota Message: Note: Quota has exceeded, however, it is not being enforced at this time but you can still make your extractions and instruments are still being counted. Please contact your Account Manager for questions. 
Manifest: #RIC,Domain,Start,End,Status,Count
Manifest: CARR.PA,Market Price,2016-09-29T15:15:00.054243730Z,2016-09-29T15:35:15.302397666Z,Active,6926
Manifest: TESTFINN=TEST,Market Price,,,Inactive,0
Manifest: US438516AC05=JPNY,Market Price,,,Inactive,0

The file contents give us important details about the extraction. It is recommended to save the extraction notes, and also to parse them to detect issues, warnings or errors.

Clean up

In the initial steps of this tutorial, we created 3 items on the DSS server: an instrument list, a report template and a schedule.

Whenever such an item is not required any more, it can be deleted, programmatically.

Deleting an item requires a DELETE method, and using the id of the item to be deleted inside the appropriate path. This is why, in the item creation steps above, we systematically saved the ids received them from the DSS server. If you did not save them, then you will have to delete them manually, using the web GUI.

Deleting unused items is a good practice, to avoid cluttering the DSS server with lots of old useless items, and is also a pre-requisite if you want to re-use a name for an instrument list, report template or schedule.

The choice of deleting an item or not depends on its future utility. Obviously, a recurring schedule that is expected to run on a regular basis will not be deleted !

In this tutorial we shall delete all the items we created. Results can be checked by using the web GUI.

Delete the schedule from the server - HTTP request

Note: the schedule id is a parameter in the path.

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/Schedules('0x07861d8f7ebc7797')

Method: DELETE

Headers:

    	
            Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Delete the schedule from the server - HTTP response

If the token and schedule id are valid, this is the response we get:

Status: 204 No Content

Body: Response does not contain any data.

Delete the report template from the server - HTTP request

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ReportTemplates('0x07861c85fc0c7796')

Method: DELETE

Headers:

    	
            Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Delete the report template from the server - HTTP response

If the token and report template id are valid, this is the response we get:

Status: 204 No Content

Body: Response does not contain any data.

Delete the instrument list from the server - HTTP request

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/InstrumentLists('0x07861e8bf35c779d')

Method: DELETE

Headers:

    	
            Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Delete the instrument list from the server - HTTP response

If the token and instrument list id are valid, this is the response we get:

Status: 204 No Content

Body: Response does not contain any data.

Error handling

We handled a few cases above, like instrument validation when appending instruments to a list, or checking the extraction notes to understand why some data is missing.

A very common error, especially when learning, is attempting to run a piece of code several times, and running into item creation errors due to existing ones with the same name. The following describes the symptoms and remedies.

Create an instrument list (or other item) error - HTTP response if duplicate

If an instrument list with the same name already exists, an error is returned:

Status: 400 Bad Request

Relevant headers:

    	
            Content-Type: application/json; charset=utf-8
X-Validation-Messages: [{"Id":"DuplicateInstrumentListName","ItemType":"Instrument List","ItemId":"0x07861e8bf35c779d","PropertyName":"Name","Severity":4,"Message":"An instrument list with the selected name myInstrumentListName already exists","DiagnosticMessage":null}]

The message is quite clear as to the cause of the error.

Body:

    	
            {
  "error": {
    "message": "Validation Error:\r\n\r\nAn instrument list with the selected name myInstrumentListName already exists"
  }
}

Here too the message is quite clear as to the cause of the error.

Other similar errors

Similar errors will arise for a report template or extraction schedule that has the same name as an existing one.

Error causes and handling

Possible causes:

The most probable cause is that you ran a piece of code that creates an item (instrument list, report template or schedule) twice. As one cannot create 2 items with the same name, an error is generated.

It could also be possible that, by pure chance, you have an instrument list, report template and / or schedule name that has exactly the same name as one of those in this tutorial.

To solve this:

If you ran a piece of code twice, you might have forgotten to save the item's id. In that case you need to delete the item, and start again:

Manually connect to the DSS web GUI, and, if they exist, manually delete any of the following items created (and not deleted) by this tutorial:
- Instrument list myInstrumentListName
- Report template myTandSTemplateName
- Schedule myImmediateSchedule
Run your code again.

If you have an instrument list, report template and / or schedule name that has exactly the same name as one of those in this tutorial:

Change the offending name in the code of this tutorial.
Run the code again.

REST API Tutorial 12: GUI control calls: immediate extract

Tutorial purpose

Table of contents

Basic operations: explanations

Retrieve the available field list from the server

Get available field list - HTTP request

Get available field list - HTTP response

Create an instrument list

Create an instrument list - HTTP request

Create an instrument list - HTTP response

Create an instrument list - check in the DSS web GUI

Append instruments to an instrument list

Append instruments to an instrument list - HTTP request

Append instruments to an instrument list - HTTP response

Append instruments to an instrument list - check in the DSS web GUI

Create a report template

Create a report template - HTTP request

Create a report template - HTTP response

Create a report template - check in the web GUI

Schedule an immediate extraction

Schedule an immediate extraction - HTTP request

Schedule an immediate extraction - HTTP response

Schedule an immediate extraction - check in the DSS web GUI

Check the extraction status

Extraction status life cycle

Check the extraction status - HTTP request

Check the extraction status - HTTP response

Report Extraction - HTTP request

Retrieve the list of extraction files - HTTP response

Retrieve the data from the server

Retrieve the data from the server - HTTP response

Retrieve the extraction notes from the server - HTTP request

Retrieve the extraction notes from the DSS server - HTTP response

Clean up

Delete the schedule from the server - HTTP request

Delete the schedule from the server - HTTP response

Delete the report template from the server - HTTP request

Delete the report template from the server - HTTP response

Delete the instrument list from the server - HTTP request

Delete the instrument list from the server - HTTP response

Error handling

Create an instrument list (or other item) error - HTTP response if duplicate

Other similar errors

Error causes and handling