| Last update | Nov 2023 |
| Environment | Any |
| Language | Any HTTP is supported |
| Compilers | None |
| Prerequisites | DSS login, internet access |
| Source code | Below |
Tutorial purpose
This tutorial goes through the basics of programmatic GUI control.
Instead of using an On Demand extraction, which is a simplified query, it performs a whole set of actions that could also be done manually in the DSS web GUI. This means that the result of the actions described in this tutorial can also be seen in the DSS web GUI, which gives you a good testing tool.
For an explanation on these two approaches, look here.
In this tutorial an EoD (End of Data) extraction is performed. Other extractions will work in a similar way.
Table of contents
- Retrieve the available field list from the DSS server
- Create an instrument list
- Append instruments to an instrument list
- Create a report template
- Schedule an immediate extraction
- Check the extraction status
- Retrieve the extraction report
- Retrieve the data from the DSS server
- Clean up
- Error handling
Notes:
- For an explanation of the three basic elements (instrument list, report template and schedule), see here.
- The first step is exactly the same as what we did at the start of Tutorial 2, but for the sake of completeness we repeat it here. All the other steps are new.
Retrieve the available field list from the DSS server
Get available field list for End of Day - HTTP request
If you do not know what field lists are available, you can request a list of those available.
The report template type must be specified in the path. In this case we want End of Day data, so we set a value of EndOfDayPricing.
Note: for all requests we need a user token. This was retrieved in Tutorial 1.
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/GetValidContentFieldTypes(ReportTemplateType=DataScope.Select.Api.Extractions.ReportTemplates.ReportTemplateTypes'EndOfDayPricing')
Method: GET
Headers:
Prefer: respond-async
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Get available field list for End of Day - HTTP response
If the token is valid, this is the response we get:
Status: 200 OK
Relevant headers:
Content-Type: application/json; charset=utf-8
Body:
There are more than 700 values in the response. Here is the beginning of the response:
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ContentFieldTypes",
"value": [
{
"Code": "EOD.3 Month High",
"Name": "3 Month High",
"Description": "Instrument's highest price over the last 3 months",
"FormatType": "Number",
"FieldGroup": " "
},
{
"Code": "EOD.3 Month Low",
"Name": "3 Month Low",
"Description": "Instrument's lowest price over the last 3 months",
"FormatType": "Number",
"FieldGroup": " "
},
{
"Code": "EOD.3 Month Percent Change",
"Name": "3 Month Percent Change",
"Description": "Percentage change in price over the last 3 months",
"FormatType": "Number",
"FieldGroup": " "
},
{
"Code": "EOD.52 Week High",
"Name": "52 Week High",
"Description": "Fund's 52-week high price represented in the fund's default currency",
"FormatType": "Number",
"FieldGroup": " "
},
{
"Code": "EOD.52 Week Low",
"Name": "52 Week Low",
"Description": "Fund's 52-week low price represented in the fund's default currency",
"FormatType": "Number",
"FieldGroup": " "
},
This goes on with all the other available fields. Here is the last part:
{
"Code": "EOD.Z Spread",
"Name": "Z Spread",
"Description": "Constant spread in basis points that will make the price equal to the present value of the cash flows when added to each relevant point of the yield curve",
"FormatType": "Number",
"FieldGroup": " "
},
{
"Code": "EOD.Zero Curve",
"Name": "Zero Curve",
"Description": "Name of Zero Curve used in OTC Pricing",
"FormatType": "Text",
"FieldGroup": " "
},
{
"Code": "EOD.ZPage",
"Name": "ZPage",
"Description": "Unique system-assigned four-digit alphanumeric identifier for Eurobonds",
"FormatType": "Text",
"FieldGroup": " "
}
]
}
The result contains the field code, name, a description, field type (number, text, date) and group. Use this to choose the field names you want. In the next step we will make a request for data, using some data fields we chose.
Create an instrument list
Create an instrument list - HTTP request
In this step we create an empty instrument list. We will populate it in the next step, by appending instrument identifiers to the list.
The body of the request must mention:
- The data type, which is an InstrumentList.
- The name of the instrument list. This must be unique, an error will be generated if the name already exists.
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/InstrumentLists
Method: POST
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Body:
{
"@odata.type": "#DataScope.Select.Api.Extractions.SubjectLists.InstrumentList",
"Name": "myInstrumentListName"
}
Create an instrument list - HTTP response
If the token is valid, and an instrument list with the same name does not yet exist, this is the response we get:
Status: 201 Created
Relevant headers:
Content-Type: application/json; charset=utf-8
Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/InstrumentLists('')
Body:
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#InstrumentLists/$entity",
"ListId": "0x080833021dedf0a1",
"Name": "myInstrumentListName",
"Count": 0,
"Created": "2022-05-30T08:01:15.544Z",
"Modified": "2022-05-30T08:01:15.544Z"
}
Your application must cache the value of ListId returned in the body content, it will be required in the next steps.
In case of error, see the Error handling section below.
Create an instrument list - check in the DSS web GUI
Let us display the result of this operation in the DSS web GUI: click on DATASCOPE SELECT top left, then on Instrument Lists:
At this stage, the instrument list is created, but when we click on it we see it is empty:
If there are many and you can’t find it, sort the entries by clicking on the Creation Date column header.
Append instruments to an instrument list
Append instruments to an instrument list - HTTP request
Intrument identifiers are added to the instrument list by appending them. This can even be done several times.
The request path refers to the instrument list Id (0x080833021dedf0a1) which was returned when we created the instrument list in the previous step.
The body of the request must mention:
- The list of instrument identifiers, each one with its type (Cusip, Isin, Ric, etc., more than 30 are available). The user defined identifier is optional; it is just an additional tag that will be stored in the instrument list on the DSS server. It could contain a customer internal identifier, asset class, industry sector, country, currency, etc. Instrument identifiers can be filtered or sorted by this tag in the DSS web GUI.
- In the example below we define one instrument using a CUSIP code (Committee on Uniform Securities Identification Procedures, an identification system for north American and Canadian stocks), another one using a RIC.
More information on them can be found by browsing in the API Reference Tree or searching in the help in the DSS web GUI:
- We also include, for training purposes, an invalid instrument, to see the effects of instrument validation.
- What to do if there is a duplicate instrument identifier.
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/InstrumentLists('0x080833021dedf0a1')/DataScope.Select.Api.Extractions.InstrumentListAppendIdentifiers
Method: POST
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Body:
{
"Identifiers": [
{
"Identifier": "IBM.N",
"IdentifierType": "Ric",
"UserDefinedIdentifier": "EQUITYTEST"
},
{
"Identifier": "438516AC0",
"IdentifierType": "Cusip",
"UserDefinedIdentifier": "BONDTEST"
},
{
"Identifier": "JUNK.JUNK",
"IdentifierType": "Ric",
"UserDefinedIdentifier": "INVALID"
}
],
"KeepDuplicates": false
}
Append instruments to an instrument list - HTTP response
If the token is valid, this is the response we get:
Status: 200 OK
Relevant headers:
Content-Type: application/json; charset=utf-8
Body:
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#DataScope.Select.Api.Extractions.SubjectLists.InstrumentsAppendIdentifiersResult",
"ValidationResult": {
"ValidInstrumentCount":2,
"OpenAccessSegments": [],
"StandardSegments": [
{
"Code": "E",
"Description": "Equity",
"Count": 1
},
{
"Code": "G",
"Description": "GORP",
"Count": 1
}
],
"ValidationDuplicates": [],
"Messages": [
{
"Severity": "Info",
"Message": "RIC, JUNK.JUNK (not found)"
}
]
},
"AppendResult": {
"AppendedInstrumentCount": 2,
"AppendDuplicates": []
}
}
The body contains several sections:
- Instrument identifiers validation results, including the number of valid instruments, and some high level information on the segmentation by asset class of the instrument identifiers.
- If there are issues, a messages section will list all invalid instruments.
- Instrument identifiers append results, including the number of appended instruments, and details on eventual duplicates,
Append instruments to an instrument list - check in the DSS web GUI
The result of this operation can be seen in the DSS web GUI. Now we find our 2 instruments in the instrument list:
Create a report template
Create a report template - HTTP request
The report template type must be set in the path of the request. In this case it is an End of Day pricing report template.
The body of the request must mention several parameters. The main ones are:
- The data type, which in this case is an EndOfDayPricingReportTemplate.
- The name of the report template. This must be unique, an error will be generated if the name already exists.
- The list of data content fields we want in the result. These were chosen among all the available ones, see Retrieve the available field list from the DSS server above.
- URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/EndOfDayPricingReportTemplates
Method: POST
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Body:
{
"@odata.type": "#DataScope.Select.Api.Extractions.ReportTemplates.EndOfDayPricingReportTemplate",
"ShowColumnHeaders": false,
"Name": "myEodTemplateName",
"Headers": [],
"Trailers": [],
"ContentFields": [
{
"FieldName": "Instrument ID"
},
{
"FieldName": "Security Description"
},
{
"FieldName": "Universal Close Price Date"
},
{
"FieldName": "Universal Close Price"
}
],
"Condition": null
}
Create a report template - HTTP response
If the token is valid, and a report template with the same name does not yet exist, this is the response we get:
Status: 201 Created
Relevant headers:
Content-Type: application/json; charset=utf-8
Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/EndOfDayPricingReportTemplates('')
Body:
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#EndOfDayPricingReportTemplates/$entity",
"ReportTemplateId": "0x0808321dd61df091",
"ShowColumnHeaders": false,
"CompressionType": "None",
"CreateDate": "2022-05-30T08:11:16.591Z",
"LastChangedDate": "2022-05-30T08:11:16.591Z",
"Name": "myEodTemplateName",
"OutputFormat": "CommaSeparatedValues",
"ReportFieldCount": 4,
"Delimiter": "None",
"DeliveryType": "None",
"TemplateTypeCode": "EOD",
"Headers": [],
"Trailers": [],
"ContentFields": [
{
"FieldName": "Instrument ID",
"Justification": "Center",
"WidthStyle": "VariableWidth",
"Format": {
"@odata.type": "#DataScope.Select.Api.Extractions.ReportTemplates.ContentFieldTextFormat",
"Capitalization": "None"
}
},
{
"FieldName": "Security Description",
"Justification": "Center",
"WidthStyle": "VariableWidth",
"Format": {
"@odata.type": "#DataScope.Select.Api.Extractions.ReportTemplates.ContentFieldTextFormat",
"Capitalization": "None"
}
},
{
"FieldName": "Universal Close Price Date",
"Justification": "Center",
"WidthStyle": "VariableWidth",
"Format": {
"@odata.type": "#DataScope.Select.Api.Extractions.ReportTemplates.ContentFieldDateFormat",
"DateFormat": "MM/dd/yyyy"
}
},
{
"FieldName": "Universal Close Price",
"Justification": "Center",
"WidthStyle": "VariableWidth",
"Format": {
"@odata.type": "#DataScope.Select.Api.Extractions.ReportTemplates.ContentFieldNumberFormat",
"DecimalPlaces": 9,
"DecimalSeparator": "Comma",
"IntegerPlaces": 18,
"UseLeadingZero": false,
"NegativeSignPosition": "Before",
"ThousandSeparator": "Space",
"UseThousandSeparator": true,
"UseTrailingZero": false
}
}
]
}
Your application must cache the value of ReportTemplateId returned in the body content, it will be required in the next steps.
In case of error, see the Error handling section below.
Create a report template - check in the DSS web GUI
Let us display the result of this operation in the DSS web GUI: click on DATASCOPE SELECT top left, then on Report Templates:
We find our report template with 4 fields:
If there are many and you can’t find it, sort the entries by clicking on the Creation Date column header.
Schedule an immediate extraction
Schedule an immediate extraction - HTTP request
An extraction schedule defines when the data will be retrieved, and triggers the actual extraction. It must refer to an instrument list (for which the extraction will be done), and a report template (defining what data needs to be extracted).
An extraction schedule can run once or recurring. Its timing can be immediate, at a fixed time, or triggered when data is available.
It is possible to define several schedules.
In this tutorial we define an extraction schedule for our previously created instrument list and report template, that will be run immediately, only once.
The body of the request must mention several parameters. The main ones are:
- The name of the extraction schedule. This must be unique, an error will be generated if the name already exists.
- The recurrence. In this example we define it to be single and immediate.
- The trigger. In this example we define an immediate trigger, with a report limited to today's data. Note that limiting a report to today's data might result in some data columns being empty, as we will see in the results.
- The instrument list id. Its value is that which was returned when we created the instrument list.
- The report template id. Its value is that which was returned when we created the report template.
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/Schedules
Method: POST
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Body:
{
"Name": "myImmediateSchedule",
"Recurrence": {
"@odata.type": "#DataScope.Select.Api.Extractions.Schedules.SingleRecurrence",
"IsImmediate": true
},
"Trigger": {
"@odata.type": "#DataScope.Select.Api.Extractions.Schedules.ImmediateTrigger",
"LimitReportToTodaysData": true
},
"ListId": "0x080833021dedf0a1",
"ReportTemplateId": "0x0808321dd61df091"
}
Schedule an immediate extraction - HTTP response
If the token is valid, and a schedule with the same name does not yet exist, this is the response we get:
Status: 201 Created
Relevant headers:
Content-Type: application/json; charset=utf-8
Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/Schedules('')
Body:
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#Schedules/$entity",
"ScheduleId": "0x0808332da48df0a2",
"Name": "myImmediateSchedule",
"TimeZone": "GMT Standard Time",
"Recurrence": {
"@odata.type": "#DataScope.Select.Api.Extractions.Schedules.SingleRecurrence",
"ExtractionDateTime": "0001-01-01T00:00:00Z",
"IsImmediate": true
},
"Trigger": {
"@odata.type": "#DataScope.Select.Api.Extractions.Schedules.ImmediateTrigger",
"LimitReportToTodaysData": true
},
"UserId": 9008895,
"CreateDate": "2022-05-30T08:18:45.455Z",
"LastChangeDate": "2022-05-30T08:18:45.455Z",
"ListId": "0x080833021dedf0a1",
"ReportTemplateId": "0x0808321dd61df091"
}
Most of the returned content is a recap of the request, but with added details. As the extraction is scheduled to be immediate, the ExtractionDateTime and TimeZone will not be used.
Your application must cache the value of ScheduleId returned in the body content, it will be required later.
In case of error, see the Error handling section below.
Schedule an immediate extraction - check in the DSS web GUI
Let us display the result of this operation in the DSS web GUI: click on DATASCOPE SELECT top left, then on Schedules:
We find our schedule, as defined. Immediately after sending the request, its status shows it has been Submitted and is Processing:
Once the scheduled data extraction has completed, its last status changes to Completed:
If there are many and you can’t find it, sort the entries by clicking on the Creation Date column header.
Check the extraction status
Extraction status life cycle
The extraction will first be queued, its status will be pending. Once triggered, it will be processed. During processing, if it is an intraday request, it might be temporarily embargoed, for the duration of the embargo delay. Once all the data is extracted, it will be complete.
The following chart illustrates the extraction events as they progress in time from left to right, with the related values of the extraction Status and DetailedStatus:
In the previous step we showed how to check the schedule and extraction status in the DSS web GUI, which is fine for testing, but it is also possible to do this programmatically.
Check the extraction status - HTTP request
The schedule id we received in the response to our schedule creation request is used as a parameter set in the path of the request.
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/Schedules('0x0808332da48df0a2')/LastExtraction
Method: GET
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Check the extraction status - HTTP response
If the token is valid, this is the response we get:
Status: 200 OK
Relevant headers:
Content-Type: application/json; charset=utf-8
Body:
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ReportExtractions/$entity",
"ReportExtractionId": "592939213",
"ScheduleId": "0x0808332da48df0a2",
"Status": "Completed",
"DetailedStatus": "Done",
"ExtractionDateUtc": "2022-05-30T08:18:45.533Z",
"ScheduleName": "myImmediateSchedule",
"IsTriggered": false,
"ExtractionStartUtc": "2022-05-30T08:18:46.000Z",
"ExtractionEndUtc": "2022-05-30T08:18:48.000Z"
}
In the response we find the Status and DetailedStatus. Here we see the status is Completed.
Your application must cache the value of ReportExtractionId returned in the body content, it will be required later.
Retrieve the extraction report
The extraction report contains the list of files generated on the DSS server.
Retrieve the extraction report - HTTP request
The report extraction id we received in the response to our extraction status request is used as a parameter set in the path of the request.
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ReportExtractions('592939213')/Files
Method: GET
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Retrieve the extraction report - HTTP response
If the token is valid, this is the response we get:
Status: 200 OK
Relevant headers:
Content-Type: application/json; charset=utf-8
Body:
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ExtractedFiles",
"value": [
{
"ExtractedFileId": "VjF8fDEwMjMzOTQ2OTA",
"ReportExtractionId": "592939213",
"ScheduleId": "0x0808332da48df0a2",
"FileType": "RicMaintenanceNote",
"ExtractedFileName": "9008895.myImmediateSchedule.20220530.091848.592939213.x01A05.ric.csv",
"LastWriteTimeUtc": "2022-05-30T08:18:48.721Z",
"ContentsExists": true,
"Size": 0,
"ReceivedDateUtc": "2022-05-30T08:18:48.721Z"
},
{
"ExtractedFileId": "VjF8fDEwMjMzOTQ2ODk",
"ReportExtractionId": "592939213",
"ScheduleId": "0x0808332da48df0a2",
"FileType": "Full",
"ExtractedFileName": "9008895.myImmediateSchedule.20220530.091848.592939213.x01A05.csv",
"LastWriteTimeUtc": "2022-05-30T08:18:48.720Z",
"ContentsExists": true,
"Size": 95,
"ReceivedDateUtc": "2022-05-30T08:18:48.720Z"
},
{
"ExtractedFileId": "VjF8fDEwMjMzOTQ2ODg",
"ReportExtractionId": "592939213",
"ScheduleId": "0x0808332da48df0a2",
"FileType": "Note",
"ExtractedFileName": "9008895.myImmediateSchedule.20220530.091848.592939213.x01A05.csv.notes.txt",
"LastWriteTimeUtc": "2022-05-30T08:18:48.725Z",
"ContentsExists": true,
"Size": 1823,
"ReceivedDateUtc": "2022-05-30T08:18:48.725Z"
}
]
}
We focus on 2 files have been generated on the DSS server:
- The Full extracted data itself, which could be delivered in one or several files, depending on the occurrence of an embargo (which could occur for an intraday request) and preference settings. As this extraction is for EOD data, the file type is Full, independent of DSS preference settings.
- The extraction Note file, which contains details of the extraction, including information on applied embargoes. To understand embargoes, and how DSS handles them, study the entire .Net SDK Tutorial 4.
Each file has an ExtractedFileId. Your application must cache the values of these file ids, to be able to access the files contents.
Retrieve the data from the DSS server
Once the status is Completed, and we have retrieved the ExtractedFileId, we can retrieve the data.
Retrieve the data from the DSS server - HTTP request
The extracted file id we received for the Full data file in the response to our extraction report request is used as a parameter set in the path of the request.
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractedFiles('VjF8fDEwMjMzOTQ2ODk')/$value
Method: GET
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Retrieve the data from the DSS server - HTTP response
If the token is valid, this is the response we get:
Status: 200 OK
Relevant headers:
Content-Encoding: gzip
Content-Type: text/plain
Note: contrary to most responses, this time the content is not in JSON format !
Body:
IBM.N,INTERNATIONAL BUSINESS MACHINES ORD,05/27/2022,
438516AC0,HON 9.500 06/01/16 MATd,,
The content of the file is the data, a CSV compressed in GZIP format. Postman automatically decompresses the data.
There are 4 values in each line, corresponding to the 4 requested fields (Instrument ID, Security Description, Universal Close Price Date, Universal Close Price). Depending on the instruments, chosen fields and time of request, it can happen that no data is available for some fields, like here for the last one.
Retrieve the extraction notes from the DSS server - HTTP request
The example above illustrated how to retrieve the contents of the data file. Retrieving the contents of the other files can be done using their respective ExtractedFileId values. The extracted file id we received for the Note file in the response to our extraction report request is used as a parameter set in the path of the request.
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractedFiles('VjF8fDEwMjMzOTQ2ODg')/$value
Method: GET
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Retrieve the extraction notes from the DSS server - HTTP response
If the token is valid, this is the response we get:
Status: 200 OK
Relevant headers:
Content-Type: text/plain
Body:
Extraction Services Version 16.0.43633 (806c08a4ae8f), Built May 9 2022 17:21:13
Holiday Rollover of Universal Close Price waived.
Processing started at 05/30/2022 09:18:48.
User ID: 9008895
Extraction ID: 592939213
Correlation ID: CiD/9008895/AAAAAA.0808332da47df0a2/RA/EXT.592939213
Schedule: myImmediateSchedule (ID = 0x0808332da48df0a2)
Input List (2 items): myInstrumentListName (ID = 080833021dedf0a1) Created: 05/30/2022 09:01:15 Last Modified: 05/30/2022 09:09:23
Schedule Time: 05/30/2022 09:18:45
Report Template (4 fields): myEodTemplateName (ID = 0x0808321dd61df091) Created: 05/30/2022 09:11:16 Last Modified: 05/30/2022 09:11:16
(CSP,438516AC0,EJV) is inactive.
Columns for (RIC,IBM.N,NYS) suppressed due to trade date other than today
Processing completed successfully at 05/30/2022 09:18:48, taking 0.641 Secs.
Extraction finished at 05/30/2022 08:18:48 UTC, with servers: x01A05, QSDHA1 (0.0 secs), QSHC18 (0.1 secs)
Usage Summary for User 9008895, Client 65508, Template Type EOD Pricing
Base Usage
Instrument Instrument Terms Price
Count Type Subtype Source Source
------- ----------------------------------- ---------------------------- -------------- ----------------------------------------
1 Corporate N/A N/A
1 Equities N/A N/A
-------
2 Total instruments charged.
0 Instruments with no reported data.
=======
2 Instruments in the input list.
No Evaluated Pricing Service complex usage to report -- 2 Instruments in the input list had no reported data.
Writing RIC maintenance report.
The file contents give us details about the extraction.
In this particular case we also find an explanation as to why some the Universal Close Price field is empty:
- The message "Columns suppressed due to trade date other than today" tells us some data was not delivered, and why.
- When we created our schedule, we set a parameter inside our trigger to limit the report to todays data: "LimitReportToTodaysData": true. That explains why we do not receive data from a previous day. Had we set that value to false, we would have received values for all data fields.
Clean up
In the initial steps of this tutorial, we created 3 items on the DSS server: an instrument list, a report template and a schedule.
Whenever such an item is not required any more, it can be deleted, programmatically.
Deleting an item requires a DELETE method, and using the id of the item to be deleted inside the appropriate path. This is why, in the item creation steps above, we systematically saved the ids received them from the DSS server. If you did not save them, then you will have to delete them manually, using the DSS web GUI.
Deleting unused items is a good practice, to avoid cluttering the DSS server with lots of old useless items, and is also a pre-requisite if you want to re-use a name for an instrument list, report template or schedule.
The choice of deleting an item or not depends on its future utility. Obviously, a recurring schedule that is expected to run on a regular basis will not be deleted !
In this tutorial we shall delete all the items we created. Results can be checked by using the DSS web GUI.
Delete the schedule from the DSS server - HTTP request
Note: the schedule id is a parameter in the path.
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/Schedules('0x0808332da48df0a2')
Method: DELETE
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Delete the schedule from the DSS server - HTTP response
If the token and schedule id are valid, this is the response we get:
Status: 204 No Content
Body: Response does not contain any data.
Delete the report template from the DSS server - HTTP request
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ReportTemplates('0x0808321dd61df091')
Method: DELETE
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Delete the report template from the DSS server - HTTP response
If the token and report template id are valid, this is the response we get:
Status: 204 No Content
Body: Response does not contain any data.
Delete the instrument list from the DSS server - HTTP request
URL:
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/InstrumentLists('0x080833021dedf0a1')
Method: DELETE
Headers:
Prefer: respond-async
Content-Type: application/json
Authorization: Token F0ABE9A3FFF2E02E10AE2765ED872C59B8CC3B40EBB61B30E295E71DE31C254B8648DB9434C2DF9299FDC668AA123501F322D99D45C8B93438063C912BC936C7B87062B0CF812138863F5D836A7B31A32DCA67EF07B3B50B2FC4978DF6F76784FDF35FCB523A8430DA93613BC5730CDC310D4D241718F9FC3F2E55465A24957CC287BDEC79046B31AD642606275AEAD76318CB221BD843348E1483670DA13968D8A242AAFCF9E13E23240C905AE46DED9EDCA9BB316B4C5C767B18DB2EA7ADD100817ADF059D01394BC6375BECAF6138C25DBA57577F0061
Delete the instrument list from the DSS server - HTTP response
If the token and instrument list id are valid, this is the response we get:
Status: 204 No Content
Body: Response does not contain any data.
Error handling
We handled a few cases above, like instrument validation when appending instruments to a list, or checking the extraction notes to understand why some data is missing.
A very common error, especially when learning, is attempting to run a piece of code several times, and running into item creation errors due to existing ones with the same name. The following describes the symptoms and remedies.
Create an instrument list (or other item) error - HTTP response if duplicate
If an instrument list with the same name already exists, an error is returned:
Status: 400 Bad Request
Relevant headers:
Content-Type: application/json; odata.metadata=minimal
X-Validation-Messages: [{"Id":"DuplicateInstrumentListName","ItemType":"Instrument List","ItemId":"0x053655def87676d7","PropertyName":"Name","Severity":4,"Message":"An instrument list with the selected name myInstrumentListName already exists","DiagnosticMessage":null}]
The message is quite clear as to the cause of the error.
Body:
{
"error": {
"message": "Validation Error:\r\n\r\nAn instrument list with the selected name myInstrumentListName already exists"
}
}
Here too the message is quite clear as to the cause of the error.
Other similar errors
Similar errors will arise for a report template or extraction schedule that has the same name as an existing one.
Error causes and handling
Possible causes:
The most probable cause is that you ran a piece of code that creates an item (instrument list, report template or schedule) twice. As one cannot create 2 items with the same name, an error is generated.
It could also be possible that, by pure chance, you have an instrument list, report template and / or schedule name that has exactly the same name as one of those in this tutorial.
To solve this:
If you ran a piece of code twice, you might have forgotten to save the item's id. In that case you need to delete the item, and start again:
- Manually connect to the DSS web GUI, and, if they exist, manually delete any of the following items created (and not deleted) by this tutorial:
- Instrument list myInstrumentListName
- Report template myEodTemplateName
- Schedule myImmediateSchedule
- Run your code again.
If you have an instrument list, report template and / or schedule name that has exactly the same name as one of those in this tutorial:
- Change the offending name in the code of this tutorial.
- Run the code again.