Testing
LSEG Tick History REST API with CURL
Introduction
In this article we will go over testing LSEG Tick History API extraction workflow with curl. Using on demand TickHistoryRawExtractionRequest as an example, we will:
- Authenticate to obtain a valid token
- Submit on-demand extraction request
- Ascertain that the extraction request processing has been completed
- Obtain the extraction result
A specific version of curl is not required to do this type of testing.
Another on-demand request can be substituted in the workflow, in place of TickHistoryRawExtractionRequest, and tested with curl.
Authentication - Obtaining A Valid Token
curl -i -X POST -k -H "Content-Type: application/json" -H "Prefer: respond-async" https://selectapi.datascope.refinitiv.com/RestApi/v1/Authentication/RequestToken -H 'X-Direct-Download: True' -x MYVALIDPROXY:MYPROXYPORT -d @creds.txt
with creds.txt file stored in the run directory and containing:
{
"Credentials": {
"Username": "MYVALIDDSSUSERID",
"Password": "MYVALIDDSSPASSWORD"
}
}
The result from this call should contain a valid token, that is good for 24 hours of use with other requests, similar to
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/10.0
BeginRequestTime: 22:21:25.9504696
BeginRequestDate: 2021-08-23
X-Request-Execution-Correlation-Id: CiD/0/AAAAAA.07ae7673cf8d0f48/RA
X-App-Id: Custom.RestApi
X-App-Version: 15.0.155.64
CPUUtilization: 14.20399
RequestsPerSec: 4.75508
W3WP-PrivateBytes: 7196
Date: Mon, 23 Aug 2021 22:21:25 GMT
Content-Length: 405
{"@odata.context":"https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#Edm.String","value":"_YuBtkhCDqU8h3S7amxPa4pDtdLKcuefiYq0ab9s23RPfXmi2c7By2ZZZA1VcnoerhtUfsSQnyBlmAN0sDyB-iXSDxm8rxefnv21c7x_tDncPB_Cnp22SZoLhiXyaqXCXmuZcuA1kX3zsyHTGQo_CM5cDzhWUdpbVeiLKpIsXfrsj47kfCTOem1w_RjXvW2G_fsdot7XDt_-krdXGF0L-mpkLtdPJPXWPoBkoDGwWAhygraUpslQy-QhUucEX5Ra8ov5f7Hhu1ZYq4LhKxHZ51aILypJy3pWefVHXqcp21Ts"}
Submit On-Demand Extraction Request
We use the valid token obtained in the previous step to submit TickHistoryRawExtractionRequest
curl -i -X POST -k -H "Content-Type: application/json" -H "Prefer: respond-async" "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/json' -H 'X-Direct-Download: True' -x MYVALIDPROXY:MYPROXYPORT -d @requestBody.txt
with file requestBody.txt stored in the run folder and containing:
{
"ExtractionRequest": {
"@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest",
"IdentifierList": {
"@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers": [{
"Identifier": "CARR.PA",
"IdentifierType": "Ric"
}]
},
"Condition": {
"MessageTimeStampIn": "GmtUtc",
"ReportDateRangeType": "Range",
"QueryStartDate": "2016-09-29T12:00:00.000Z",
"QueryEndDate": "2016-09-29T12:10:00.000Z",
"ExtractBy": "Ric",
"SortBy": "SingleByRic",
"DomainCode": "MarketPrice",
"DisplaySourceRIC": true
}
}
}
In result we look for two pieces for information, status and ExtractionID:
HTTP/1.1 202 Accepted
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRawResult(ExtractionId='0x07ae64718b9d0f41')
Server: Microsoft-IIS/10.0
BeginRequestTime: 21:30:25.3627291
BeginRequestDate: 2021-08-23
Status: InProgress
Progress:
X-Request-Execution-Correlation-Id: CiD/19945/AAAAAA.07ae64718bad0f41/RA
X-App-Id: Custom.RestApi
X-App-Version: 15.0.155.64
CPUUtilization: 26.47529
RequestsPerSec: 0
W3WP-PrivateBytes: 7120
Date: Mon, 23 Aug 2021 21:30:55 GMT
Content-Length: 0
The status code of 202 means that request is valid and is accepted for processing, the status code of 200 means that request is valid, and request processing is completed. Any other status means that request should be examined as it is not valid in some aspect of the other.
The extraction Id will be used in the next steps to retrieve the results, as well as examine the status of the request until it's processing is completed, resulting in status of 200.
Checking Status of the Extraction Using Extraction Id
This step is required if the result from the previous step is not completed = 200 but is accepted = 202.
Next we use the extraction Id just obtained to check on the status of the request, waiting in between the checks, until the status that is returned is completed = 200
curl -i -X GET -k -H "Content-Type: application/json" -H "Prefer: respond-async" "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRawResult(ExtractionId='0x07ae64718b9d0f41')" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/json' -H 'X-Direct-Download: True' -x MYPROXY:MYPROXYPORT
the result being the status of the request:
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/10.0
BeginRequestTime: 21:57:10.4224282
BeginRequestDate: 2021-08-23
X-Request-Execution-Correlation-Id: CiD/19945/AAAAAA.07ae85c312ed0f67/RA
X-App-Id: Custom.RestApi
X-App-Version: 15.0.155.64
CPUUtilization: 17.59716
RequestsPerSec: 7.619745
W3WP-PrivateBytes: 1569704
Date: Mon, 23 Aug 2021 21:57:10 GMT
Content-Length: 1414
Obtaining a Ready Extraction Result
Once the result is ready we can GET the result and store it into a file
curl -X GET "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x07ae64718b9d0f41')/$value" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/gzip' -H 'X-Direct-Download: True' -x MYVALIDPROXY:MYPROXYPORT -v -N -o "tst.csv.gz"
We can next unzip the contents of file tst.csv.gz:
#RIC,Domain,Date-Time,GMT Offset,Type,MsgClass/FID number,UpdateType/Action,FID Name,FID Value,FID Enum String,PE Code,Template Number,Key/Msg Sequence Number,Alias Underlying RIC,Number of FIDs
CARR.PA,Market Price,2016-09-29T12:00:00.942730204Z,+2,Raw,UPDATE,UNSPECIFIED,,,,8132,,2800,,13
,,,,FID,4981,,TRTN,0.89,
,,,,FID,4982,,TRTN_3MT,5.85,
,,,,FID,5615,,YR_TRTN,-5.23,
,,,,FID,5619,,YTD_TRTN,-7.95,
,,,,FID,9257,,ASY_VA_DT,2016-09-29,
,,,,FID,9959,,TRTN_1W,-0.59,
,,,,FID,9960,,TRTN_1M,4.97,
,,,,FID,9961,,TRTN_2Y,1.34,
,,,,FID,9962,,TRTN_3Y,-0.39,
,,,,FID,9963,,TRTN_4Y,60.61,
,,,,FID,9964,,TRTN_5Y,52.58,
,,,,FID,9965,,MTD_TRTN,2.29,
,,,,FID,9966,,QTD_TRTN,3.29,
CARR.PA,Market Price,2016-09-29T12:00:03.161984291Z,+2,Raw,UPDATE,QUOTE,,,,8132,,2816,,14
,,,,FID,3866,,AUC_BID,,
,,,,FID,3862,,AUC_BIDSIZ,,
,,,,FID,22,,BID,23.24,
,,,,FID,6579,,BID_COND_N,0,
,,,,FID,30,,BIDSIZE,2752,
Another, very popular approach, that is used to test LSEG Tick History and Datascope Select REST APIs is tool Postman. Detailed info on testing with Postman complete with starter example deck can be found in REST API Tutorials mentioned in References section