| Last update | Nov 2023 |
| Environment | Any |
| Language | Any HTTP is supported |
| Compilers | None |
| Prerequisites | DSS login, internet access |
| Source code | Below |
This tutorial explains how to:
The On Demand extractions in tutorials 2-8 use the ExtractWithNotes endpoint, whereas this tutorial uses the ExtractRaw endpoint, in a use case similar to that in tutorial 4.
Instead of delivering data in JSON format, this endpoint delivers data in a Gzipped CSV format. This is usefull for large data requests. For more information on these two endpoints, please refer to the Tutorials introduction.
Important note:
The legacy Timeseries Pricing extraction will be removed end February 2020, the more powerful Price History extraction replaces it. This tutorial was updated in May 2019 to reflect this. If you are using the Timeseries Pricing call, you must modify your code. As part of the migration process, it is important to understand that, in some instances, field content extracted from the Timeseries Pricing report will be available under different field names in the Price History report. It is also possible that some previously populated values in the Timeseries Pricing report will be blank in the Price History report. This is due to data mapping changes. For more information, please refer to the Product Change Notification PCN 10897.
If you do not know what fields are available, you can request a list of them. For a price history request this was covered in Tutorial 4.
The On Demand extractions in tutorials 2-8 use the ExtractWithNotes endpoint, whereas this tutorial uses the ExtractRaw endpoint, in a use case similar to that in tutorial 4. Instead of delivering data in JSON format, this endpoint delivers data in a Gzipped CSV format.
Apart from the endpoint, this request has the same format as that in Tutorial 4. Please refer to that tutorial for the details.
Note: the tutorial 14 sample in the Postman collection retrieves data for 2 RICs from 1996 till end 2017, whereas tutorial 4 requests 3 RICs for a short period, but those are the only differences.
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw
This endpoint URL is different from those we used previously !
Method: POST
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Body:
{
"ExtractionRequest": {
"@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.PriceHistoryExtractionRequest",
"ContentFieldNames": [
"File Code",
"RIC",
"Trade Date",
"Last Trade Price",
"Universal Close Price",
"Alternate Close Price",
"High Price",
"Low Price",
"Open Price",
"Volume Weighted Average Price",
"Turnover",
"Volume",
"Accumulated Volume Unscaled",
"Bid Price",
"Asset Type",
"Quote ID",
"Bid Yield",
"Exchange Code",
"Currency Code"
],
"IdentifierList": {
"@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers": [
{ "Identifier": "ALVG.DE", "IdentifierType": "Ric" },
{ "Identifier": "IBM.N", "IdentifierType": "Ric" }
]
},
"Condition": {
"AdjustedPrices": true,
"QueryStartDate": "1996-01-01T00:00:00.000Z",
"QueryEndDate": "2017-12-31T23:59:59.999Z"
}
}
}
Note: an embargo can not occur as this is historical data, not intraday data.
On Demand extraction requests are executed as soon as possible. There is no guarantee on the delivery time, it depends on the amount of requested data, and the server load.
In the request we set a preference for an asynchronous response. We will get a response in 30 seconds (default wait time) or less.
The HTTP status of the response can have one of several values, here we shall detail the most likely ones:
It is strongly recommended that you ensure your code handles all possible status codes.
When requests take more than 30 seconds, a 202 Accepted is returned as the first response. This is fairly common, which means that 202 Accepted will be the usual first response.
You can customize the wait time, but this is not recommended.
Let us now look at the two most common responses in detail.
The request was accepted, but processing has not yet completed. This response is the most likely, especially if the request is for a large amount of data.
Status: 202 Accepted
Relevant headers:
Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRawResult(ExtractionId='0x080832c6ba5df09d')
The location URL must be saved, we will use it in the next step, check request status. Note: the last part of the URL (0x080832c6ba5df09d) is the job ID for this request.
Body: Response does not contain any data
Instead of a 202 Accepted, we could receive a 200 OK. This means the request has completed.
Status: 200 OK
Relevant headers:
Content-Type: application/json; charset=utf-8
Body:
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#RawExtractionResults/$entity",
"JobId": "0x080832c6ba5df09d",
"Notes": [
"Extraction Services Version 16.0.43633 (806c08a4ae8f), Built May 9 2022 17:21:07\r\nProcessing started at 05/30/2022 10:02:50.\r\nUser ID: 9008895\r\nExtraction ID: 2000000402735853\r\nCorrelation ID: CiD/9008895/AAAAAA.080832c6ba4df09d/RA\r\nSchedule: 0x080832c6ba5df09d (ID = 0x0000000000000000)\r\nInput List (2 items): (ID = 0x080832c6ba5df09d) Created: 05/30/2022 10:02:50 Last Modified: 05/30/2022 10:02:50\r\nReport Template (19 fields): _OnD_0x080832c6ba5df09d (ID = 0x080832c6ba6df09d) Created: 05/30/2022 10:02:49 Last Modified: 05/30/2022 10:02:49\r\nSchedule dispatched via message queue (0x080832c6ba5df09d)\r\nSchedule Time: 05/30/2022 10:02:50\r\nTimeseries Date Range: 01/01/1996 to 12/31/2017\r\nProcessing completed successfully at 05/30/2022 10:02:54, taking 3.813 Secs.\r\nExtraction finished at 05/30/2022 09:02:54 UTC, with servers: x05q23, ETS (0.7 secs), QSDHA1 (0.0 secs), QSHC13 (0.1 secs)\r\nUsage Summary for User 9008895, Client 65508, Template Type Price History\r\nBase Usage\r\n Instrument Instrument Terms Price\r\n Count Type Subtype Source Source\r\n------- ----------------------------------- ---------------------------- -------------- ----------------------------------------\r\n 2 Equities N/A N/A\r\n-------\r\n 2 Total instruments charged.\r\n 0 Instruments with no reported data.\r\n=======\r\n 2 Instruments in the input list.\r\nNo Evaluated Pricing Service complex usage to report -- 2 Instruments in the input list had no reported data.\r\nWriting RIC maintenance report.\r\n",
"Identifier,IdentType,Source,RIC,RecordDate,MaintType,OldValue,NewValue,Factor,FactorType\r\n"
]
}
The JobId value must be saved, we will use it in the next step.
The Notes contain information on the request, IDs, timestamps, eventual errors, and extraction quota status. If the request completed successfully, it will also contain the message: Processing completed successfully.
As the request status has been returned directly (because it was a very quick extraction), the next step (check request status) is not required.
We skip it to go directly to retrieve the data, using the returned JobId.
Skip this step if the previous step returned an HTTP status of 200 OK.
If the previous step returned an HTTP status of 202 Accepted, this step must be executed, and repeated in a polling loop until it returns an HTTP status of 200 OK.
URL:
This is the Location URL, taken from the 202 response header received in the previous step.
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRawResult(ExtractionId='0x080832c6ba5df09d')
Method: GET
Headers:
This is the same as for the other steps:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
If you receive an HTTP status 202 Accepted response (the same as in the previous step), it means the request has not yet completed. You must wait a bit and check the request status again. There is no sense in polling every second, it would just place useless burden on the network and servers. Polling should be done at reasonable intervals, like 30 seconds or more (the larger the request, the longer the recommended polling interval).
If you receive an HTTP status 200 OK response, the request has completed:
Status: 200 OK
Relevant headers:
Content-Type: application/json; charset=utf-8
Body:
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#RawExtractionResults/$entity",
"JobId": "0x080832c6ba5df09d",
"Notes": [
"Extraction Services Version 16.0.43633 (806c08a4ae8f), Built May 9 2022 17:21:07\r\nProcessing started at 05/30/2022 10:02:50.\r\nUser ID: 9008895\r\nExtraction ID: 2000000402735853\r\nCorrelation ID: CiD/9008895/AAAAAA.080832c6ba4df09d/RA\r\nSchedule: 0x080832c6ba5df09d (ID = 0x0000000000000000)\r\nInput List (2 items): (ID = 0x080832c6ba5df09d) Created: 05/30/2022 10:02:50 Last Modified: 05/30/2022 10:02:50\r\nReport Template (19 fields): _OnD_0x080832c6ba5df09d (ID = 0x080832c6ba6df09d) Created: 05/30/2022 10:02:49 Last Modified: 05/30/2022 10:02:49\r\nSchedule dispatched via message queue (0x080832c6ba5df09d)\r\nSchedule Time: 05/30/2022 10:02:50\r\nTimeseries Date Range: 01/01/1996 to 12/31/2017\r\nProcessing completed successfully at 05/30/2022 10:02:54, taking 3.813 Secs.\r\nExtraction finished at 05/30/2022 09:02:54 UTC, with servers: x05q23, ETS (0.7 secs), QSDHA1 (0.0 secs), QSHC13 (0.1 secs)\r\nUsage Summary for User 9008895, Client 65508, Template Type Price History\r\nBase Usage\r\n Instrument Instrument Terms Price\r\n Count Type Subtype Source Source\r\n------- ----------------------------------- ---------------------------- -------------- ----------------------------------------\r\n 2 Equities N/A N/A\r\n-------\r\n 2 Total instruments charged.\r\n 0 Instruments with no reported data.\r\n=======\r\n 2 Instruments in the input list.\r\nNo Evaluated Pricing Service complex usage to report -- 2 Instruments in the input list had no reported data.\r\nWriting RIC maintenance report.\r\n",
"Identifier,IdentType,Source,RIC,RecordDate,MaintType,OldValue,NewValue,Factor,FactorType\r\n"
]
}
The JobId value must be saved, we will use it in the next step.
The Notes contain information on the request, IDs, timestamps, eventual errors, and extraction quota status. If the request completed successfully, it will also contain the message: Processing completed successfully.
We can now retrieve the data, using the returned JobId.
Note: this 200 response is identical in format and content to the 200 OK response that could also have been returned directly when we requested the extraction.
It is mandatory to have received a 200 OK response with a JobID from a previous step before proceeding with this last step.
URL:
Note the JobId value used as parameter in the URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x080832c6ba5df09d')/$value
Method: GET
Headers:
This is the same as for the other steps, except for the Content-Type. You must include “Accept-Encoding: gzip” for raw stream downloads, to ensure the server sends the data in Gzipped CSV format.
Prefer: respond-async
Content-Type: Accept-Encoding: gzip
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
We should get a response of this type:
Status: 200 OK
Relevant headers:
Content-Encoding: gzip
Content-Type: text/plain
Body:
The content is compressed plain text in CSV format. Depending on the nature of the data, the time range and number of instruments, the response can be quite long and contain thousands of lines.
Here is the beginning of the response content for our request (note that Postman will decompress it automatically):
File Code,RIC,Trade Date,Last Trade Price,Universal Close Price,Alternate Close Price,High Price,Low Price,Open Price,Volume Weighted Average Price,Turnover,Volume,Accumulated Volume Unscaled,Bid Price,Asset Type,Quote ID,Bid Yield,Exchange Code,Currency Code
62,ALVG.DE,1996/01/02,,130.954631,,131.068248,127.75064,127.75064,,,,294759.594399,130.954631,EQTY,0x0003dc004aeb7d03,,GER,EUR
62,ALVG.DE,1996/01/03,,131.931735,,132.613435,130.977354,131.113695,,,,343136.169053,131.931735,EQTY,0x0003dc004aeb7d03,,GER,EUR
62,ALVG.DE,1996/01/04,,131.481813,,132.340755,130.6615,131.749948,,,,160880.236638,131.431821,EQTY,0x0003dc004aeb7d03,,GER,EUR
62,ALVG.DE,1996/01/05,,130.022974,,130.795568,129.341274,130.613781,,,,268883.752143,129.841187,EQTY,0x0003dc004aeb7d03,,GER,EUR
62,ALVG.DE,1996/01/08,,131.409098,,131.795395,129.300372,130.613781,,,,307134.997218,131.113695,EQTY,0x0003dc004aeb7d03,,GER,EUR
62,ALVG.DE,1996/01/09,,131.886288,,132.704329,131.522715,132.704329,,,,329635.729615,131.795395,EQTY,0x0003dc004aeb7d03,,GER,EUR
Note: returned results will not necessarily include data for all dates you requested, it could be missing on some days, and this could vary depending on the instrument. Possible reasons: