REST API Tutorial 10: On Demand Historical Reference extraction

 

Last update Dec 2023
Environment Any
Language Any HTTP is supported
Compilers None
Prerequisites DSS login, internet access
Source code Below

Tutorial purpose

This tutorial explains how to:

  • Retrieve the available field list from the server.
  • Retrieve historical reference data from the server, using an on demand request. In this example we retrieve a few fields. The use case is to cross reference instrument codes, i.e. retrieve several instrument codes (RIC; Cusip, ISIN, Sedol, etc), for a list of instruments (with different instrument codes). For each input instrument code (ISIN, etc.) that corresponds to several RICs, all RICs are returned.
  • In this tutorial, the instrument list includes historical instruments, which are no longer quoted. For that reason we also set instrument validation options, to ensure we get data for the historical instruments as well. 

 

Table of contents


Get available field list for historical reference data - HTTP request

This step is optional. If you do not know what content field names are available, you can request a list of those available.

URL:

The available field set depends on the type of data you want to request. The data type must therefore be specified in the request, which is done by setting a report template type in the URL. As we want corporate actions data we set a value of TermsAndConditions. The preceding and following tutorials show other possibilities.

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/GetValidContentFieldTypes(ReportTemplateType=DataScope.Select.Api.Extractions.ReportTemplates.ReportTemplateTypes'HistoricalReference')
        
        
    

Method:          GET

Headers:

Note: for all requests we need a user token. This was retrieved in Tutorial 1.

    	
            

Prefer: respond-async

Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

 

Get available field list for historical reference data - HTTP response

If the token is valid, this is the response we get:

Status:                        200 OK

Relevant headers:

    	
            Content-Type: application/json; charset=utf-8
        
        
    

Body:

There are more than 50 values in the response. Here is the beginning of the response:

    	
            

{

    "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ContentFieldTypes",

    "value": [

       ...

        {

            "Code": "HRD.Australia Code",

            "Name": "Australia Code",

            "Description": "Code for the Australian Stock Exchange.",

            "FormatType": "Text",

            "FieldGroup": " "

        },

        {

            "Code": "HRD.Belgian Code",

            "Name": "Belgian Code",

            "Description": "Issue level code used for trading in Belgium.",

            "FormatType": "Text",

            "FieldGroup": " "

        },

        {

            "Code": "HRD.CESR Average Daily Turnover",

            "Name": "CESR Average Daily Turnover",

            "Description": "Average daily turnover of the security as supplied by CESR.",

            "FormatType": "Number",

            "FieldGroup": " "

        },

        {

            "Code": "HRD.CESR Average Daily Turnover Currency Code",

            "Name": "CESR Average Daily Turnover Currency Code",

            "Description": "Code indicating the currency of the security's average daily turnover as supplied by CESR.",

            "FormatType": "Text",

            "FieldGroup": " "

        },

This goes on with all the other available fields. Here is the last part:

    	
            

    {

      "Code": "HRD.User Defined Identifier6",

      "Name": "User Defined Identifier6",

      "Description": "Optional content that you can include for each instrument in your instrument list; Field can contain any information that you want to identify the instrument.",

      "FormatType": "Text",

      "FieldGroup": " "

    },

    {

      "Code": "HRD.Valoren",

      "Name": "Valoren",

      "Description": "Issue-level code used for identifying Swiss securities.",

      "FormatType": "Text",

      "FieldGroup": " "

    },

    {

      "Code": "HRD.Wertpapier",

      "Name": "Wertpapier",

      "Description": "Issue-level code used for identifying instruments in Germany.",

      "FormatType": "Text",

      "FieldGroup": " "

    }

  ]

}

The result contains the field code, name, a description, field type (number, text, date) and group.  Use this to choose the field names you want. In the next step we will make a request for data, using some data fields we chose.

 

Get historical reference data - HTTP request

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractWithNotes
        
        
    

Method:          POST

Headers:

    	
            

Prefer: respond-async

Content-Type: application/json

Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

Body:

The body of the request must mention it is an extraction request. It contains several parts:

  • The type of extraction: as we want composite data we set a value of HistoricalReferenceExtractionRequest. Preceding On Demand extraction tutorials show other possibilities.
  • The list of field names: these were determined in the first step of this tutorial.
  • The list of instrument identifiers, each one with its type. Below we define several instruments, using different identifier types. We also define validation options, to allow for historical instruments (if we did not do that, no data would be returned for instruments that are no longer quoted).
  • The conditions include the date range.
    	
            

{

    "ExtractionRequest": {

        "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.HistoricalReferenceExtractionRequest",

        "ContentFieldNames": [

           "RIC", "CUSIP", "ISIN", "SEDOL", "Issuer OrgID", "Exchange Code", "Currency Code", "Change Date"

        ],

        "IdentifierList": {

            "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",

            "InstrumentIdentifiers": [

                { "Identifier": "00209tab1", "IdentifierType": "Cusip" },

                { "Identifier": "IBM.N", "IdentifierType": "Ric" },

                { "Identifier": "US4592001014", "IdentifierType": "Isin" },

                { "Identifier": "B1YW440", "IdentifierType": "Sedol" }

            ],

            "ValidationOptions": {"AllowHistoricalInstruments": true},

            "UseUserPreferencesForValidationOptions": false

        },

        "Condition": {

            "ReportDateRangeType": "Range",

            "QueryStartDate": "2017-05-01",

            "QueryEndDate": "2017-05-31"

        }

    }

}

 

Get historical reference data - HTTP response

Request accepted, no timeout

If the token is valid, and there is no timeout (these requests can take time as they generate very large data sets), the data will be delivered.

This is the response we get:

Status:                        200 OK

Relevant headers:

    	
            Content-Type: application/json; charset=utf-8
        
        
    

Body: 

Here is the beginning of the contents:

    	
            

{

    "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#DataScope.Select.Api.Extractions.ExtractionRequests.ExtractionResult",

    "Contents": [

        {

            "IdentifierType": "Cusip",

            "Identifier": "00209TAB1",

            "RIC": "US00209TAB17=WFTI",

            "CUSIP": null,

            "ISIN": null,

            "SEDOL": null,

            "Issuer OrgID": null,

            "Exchange Code": null,

            "Currency Code": null,

            "Change Date": null

        },

        {

            "IdentifierType": "Cusip",

            "Identifier": "00209TAB1",

            "RIC": "00209TAB1=",

            "CUSIP": null,

            "ISIN": null,

            "SEDOL": null,

            "Issuer OrgID": null,

            "Exchange Code": null,

            "Currency Code": null,

            "Change Date": null

        },

        {

            "IdentifierType": "Cusip",

            "Identifier": "00209TAB1",

            "RIC": "00209TAB1=1M",

            "CUSIP": null,

            "ISIN": null,

            "SEDOL": null,

            "Issuer OrgID": null,

            "Exchange Code": null,

            "Currency Code": null,

            "Change Date": null

        },

        {

            "IdentifierType": "Cusip",

            "Identifier": "00209TAB1",

            "RIC": "00209TAB1=2M",

            "CUSIP": null,

            "ISIN": null,

            "SEDOL": null,

            "Issuer OrgID": null,

            "Exchange Code": null,

            "Currency Code": null,

            "Change Date": null

        },

        {

            "IdentifierType": "Cusip",

            "Identifier": "00209TAB1",

            "RIC": "00209TAB1=FINR",

            "CUSIP": null,

            "ISIN": null,

            "SEDOL": null,

            "Issuer OrgID": null,

            "Exchange Code": null,

            "Currency Code": null,

            "Change Date": null

        },

This goes on with all the other available fields. Here is the last part:

    	
            

        {

            "IdentifierType": "Sedol",

            "Identifier": "B1YW440",

            "RIC": "IIIl.ED",

            "CUSIP": null,

            "ISIN": "GB00B1YW4409",

            "SEDOL": "B1YW440",

            "Issuer OrgID": "68504",

            "Exchange Code": "EQD",

            "Currency Code": "GBp",

            "Change Date": "2017-05-01"

        },

        {

            "IdentifierType": "Sedol",

            "Identifier": "B1YW440",

            "RIC": "IIIl.SIG",

            "CUSIP": null,

            "ISIN": "GB00B1YW4409",

            "SEDOL": "B1YW440",

            "Issuer OrgID": "68504",

            "Exchange Code": "SIG",

            "Currency Code": "GBp",

            "Change Date": "2017-05-01"

        },

        {

            "IdentifierType": "Sedol",

            "Identifier": "B1YW440",

            "RIC": "IIIl.TQ",

            "CUSIP": null,

            "ISIN": "GB00B1YW4409",

            "SEDOL": "B1YW440",

            "Issuer OrgID": "68504",

            "Exchange Code": "TRQ",

            "Currency Code": "GBp",

            "Change Date": "2017-05-01"

        }

    ],

    "Notes": [

        "Extraction Services Version 14.5.42294 (737b0965c07f), Built Apr  8 2021 13:46:52\r\nProcessing started at 04/20/2021 04:45:26.\r\nUser ID: 9008895\r\nExtraction ID: 2000000250000838\r\nSchedule: 0x078617af43ac778c (ID = 0x0000000000000000)\r\nInput List (4 items):  (ID = 0x078617af43ac778c) Created: 04/20/2021 04:45:26 Last Modified: 04/20/2021 04:45:26\r\nReport Template (14 fields): _OnD_0x078617af43ac778c (ID = 0x078617af43cc778c) Created: 04/20/2021 04:45:22 Last Modified: 04/20/2021 04:45:22\r\nSchedule dispatched via message queue (0x078617af43ac778c)\r\nSchedule Time: 04/20/2021 04:45:24\r\nHistorical Reference Date Range: 05/01/2017 to 05/31/2017\r\n(CSP,00209TAB1,,00209TAB1=1M)  row suppressed for lack of 'SDTRRTUSCRE (MarketAxess Real-Time Datafeed US Credit), SDMARKETTRRT (MarketAxess Real-Time Datafeed -TRRT)' permission.\r\n(CSP,00209TAB1,,00209TAB1=2M)  row suppressed for lack of 'SDTRRTUSCRE (MarketAxess Real-Time Datafeed US Credit), SDMARKETTRRT (MarketAxess Real-Time Datafeed -TRRT)' permission.\r\n(ISN,US4592001014,,IBMEUR.STp)  row suppressed for lack of 'SDAACPRDN2 (Argus China Petroleum), SDAGBASN2 (Argus Base Oils), SDARGCAMAEN (Argus Caspian Markets (English)), SDAAGMKTN2 (Argus Global Markets)' permission.\r\nProcessing completed successfully at 04/20/2021 04:45:38, taking 11.759 Secs.\r\nExtraction finished at 04/20/2021 04:45:38 UTC, with servers: x03q15, DHS (11.3 secs), QSHC19 (0.1 secs), QSSHA1 (0.1 secs)\r\nWriting RIC maintenance report.\r\n",

        "Identifier,IdentType,Source,RIC,RecordDate,MaintType,OldValue,NewValue,Factor,FactorType\r\n"

    ]

}

 

Request accepted, but with a timeout

If the token is valid, but there was a timeout, this is the response we get:

Status:                        202 Accepted

Relevant headers:

    	
            Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractResult(ExtractionId='0x05bdc1da066b3036')
        
        
    

Body:

    	
            Response does not contain any data.
        
        
    

You will have to send another HTTP GET request to retrieve the result, using the location URL that was delivered in the HTTP 202 Response header.

 

 

Get historical reference data, subsequent request after a timeout - HTTP request

This is how we resend our request. Note the path, taken from the 202 response header:

URL:

    	
            https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractResult(ExtractionId='0x05bdc1da066b3036')
        
        
    

Method:          GET

Headers:

    	
            

Prefer: respond-async

Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061

If the data is available, you will get the results. Otherwise you will have to try again later. The more data is requested, the longer it can take for it to be available.