Author:
DataScope Select is a hosted delivery platform for non-streaming cross-asset data. It is also the same platform that supports LSEG Tick History. Users can extract data from DSS or LSEG Tick History via the DSS Web GUI or REST API.
Data can be extracted through either of two basic mechanisms. The first one is Scheduled Extractions, which occur at a pre-defined moment in time which could be once-of, or recurring. Scheduled extractions can be created using the website, or REST API. The second type of extraction mechanism is OnDemand extractions, which occur immediately when requested. On-Demand extractions can only be created using the REST API. The choice of mechanisms depends on the workflow you want to create.
Typically, users can use the DSS web GUI to download extraction files created by scheduled extractions via the DSS web GUI. This article explains the steps to use the DSS REST API to download extraction files created by scheduled extractions.
Scheduled Extractions
Users can use scheduled extractions to extract data regularly at a specified time or as a result of pricing availability. Users need to specify an instrument list and a report template when creating a schedule extraction. After that, users can access the DSS website to download extracted files created by scheduled extractions.
However, users can also use DSS REST API to retrieve extraction files created by scheduled extractions.
Steps to download extraction files by a schedule name via DSS REST API
There are four steps to download extraction files created by a schedule name via the DSS REST API.
1. Get a schedule ID from a schedule name
A schedule ID is a unique ID for a schedule name. In the DSS web GUI, A schedule name is used to refer to a schedule on the DSS server. However, the DSS REST API uses a schedule ID instead of a schedule name when referring to a schedule on the DSS server. Users can use the following endpoint with the HTTP GET method to get a schedule ID by its name.
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ScheduleGetByName(ScheduleName='DailySchedule')
For example, the above endpoint gets a schedule ID of a schedule named ‘DailySchedule’ on the DSS server.
If a schedule is available on the DSS server, an HTTP response with the status 200 (OK) will contain schedule information including a schedule ID. A schedule ID is in the ScheduleId field. This schedule ID will be used in the next steps. Moreover, it can be saved for future use.
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#Schedules/$entity",
"ScheduleId": "0x07b1b9370a7d1892",
"Name": "DailySchedule",
"OutputFileName": "DailyOutput%D-%T",
"TimeZone": "GMT Standard Time",
"Recurrence": {
"@odata.type": "#DataScope.Select.Api.Extractions.Schedules.WeeklyRecurrence",
"Days": [
"Monday",
"Tuesday",
"Wednesday",
"Thursday",
"Friday",
"Saturday"
]
},
"Trigger": {
"@odata.type": "#DataScope.Select.Api.Extractions.Schedules.TimeTrigger",
"LimitReportToTodaysData": false,
"At": [
{
"Hour": 17,
"Minute": 0
}
]
},
"UserId": xxx,
"CreateDate": "2021-09-03T02:32:16.560Z",
"LastChangeDate": "2021-10-08T05:17:02.713Z",
"ListId": "0x079cd588533cdc57",
"ReportTemplateId": "0x0783809e1d276ea1"
}
On the other hand, if a schedule is not available on the DSS server, an HTTP response with the status 404 (Not Found) will return.
Users can list all schedules available on the DSS server through the following endpoint with the HTTP GET method.
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/Schedules
The retrieved “ScheduleId” will be used in the next step to get completed report extractions of that schedule.
2. Get completed report extractions by a ScheduleId
Typically, a schedule extraction can have many report extractions. For example, a schedule extraction that executes daily will generate report extractions every day. The following endpoint with the HTTP GET method can be used to get the list of completed extractions for a schedule.
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ReportExtractionGetCompletedByScheduleId(ScheduleId='0x07b1b9370a7d1892')
'0x07b1b9370a7d1892’ is the ScheduleId retrieved from the previous step.
The response contains the list of completed extractions for this schedule.
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ReportExtractions",
"value": [
{
"ReportExtractionId": "548722918",
"ScheduleId": "0x07b1b9370a7d1892",
"Status": "Completed",
"DetailedStatus": "Done",
"ExtractionDateUtc": "2021-10-16T16:00:00.000Z",
"ScheduleName": "DailySchedule",
"IsTriggered": false,
"ExtractionStartUtc": "2021-10-16T16:00:01.000Z",
"ExtractionEndUtc": "2021-10-16T16:00:02.000Z"
},
{
"ReportExtractionId": "548722917",
"ScheduleId": "0x07b1b9370a7d1892",
"Status": "Completed",
"DetailedStatus": "Done",
"ExtractionDateUtc": "2021-10-15T16:00:00.000Z",
"ScheduleName": "DailySchedule",
"IsTriggered": false,
"ExtractionStartUtc": "2021-10-15T16:00:02.000Z",
"ExtractionEndUtc": "2021-10-15T16:00:03.000Z"
},
…
Otherwise, the following endpoint with the HTTP GET method can be used to get the last completed report extraction for a schedule.
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/Schedules('0x07b1b9370a7d1892')/LastExtraction
'0x07b1b9370a7d1892’ is the ScheduleId retrieved from the previous step.
The response contains the last completed report extraction for this schedule.
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ReportExtractions/$entity",
"ReportExtractionId": "548722918",
"ScheduleId": "0x07b1b9370a7d1892",
"Status": "Completed",
"DetailedStatus": "Done",
"ExtractionDateUtc": "2021-10-16T16:00:00.000Z",
"ScheduleName": "DailySchedule",
"IsTriggered": false,
"ExtractionStartUtc": "2021-10-16T16:00:01.000Z",
"ExtractionEndUtc": "2021-10-16T16:00:02.000Z"
}
Each report extraction has a “ReportExtractionId” which is used to uniquely identify a report extraction.
This “ReportExtractionId” will be used in the next step to get the list of files available in a report extraction.
3. Get a list of files available in a report extraction
Typically, each report extraction contains three files.
· A RIC Maintenance Report file lets users view descriptive and reference data fact changes to the equity-related RIC identifiers in instrument lists. This optional report identifies RIC maintenance changes that have occurred over the past ten days and is available for equities, warrants, funds, derivatives, and money/foreign exchange.
· An Extraction Notes file is generated with an extraction. It provides information about the extraction execution, including processing statistics, embargo delays, suppressed items, and warnings.
· A Full Extraction Result file contains extracted results.
The following endpoint can be used with the HTTP GET method to get a list of files available in a report extraction.
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ReportExtractions('548722918')/Files
'548722918' is the ReportExtractionId retrieved from the previous step.
The response contains a list of of files available in that report extraction.
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ExtractedFiles",
"value": [
{
"ExtractedFileId": "VjF8fDkwMjQzODM5Ng",
"ReportExtractionId": "548722918",
"ScheduleId": "0x07b1b9370a7d1892",
"FileType": "RicMaintenanceNote",
"ExtractedFileName": "DailyOutput20211016-170001.ric.csv",
"LastWriteTimeUtc": "2021-10-16T16:00:01.998Z",
"ContentsExists": true,
"Size": 90,
"ReceivedDateUtc": "2021-10-16T16:00:01.998Z"
},
{
"ExtractedFileId": "VjF8fDkwMjQzODM5NQ",
"ReportExtractionId": "548722918",
"ScheduleId": "0x07b1b9370a7d1892",
"FileType": "Full",
"ExtractedFileName": "DailyOutput20211016-170001.csv",
"LastWriteTimeUtc": "2021-10-16T16:00:01.995Z",
"ContentsExists": true,
"Size": 708,
"ReceivedDateUtc": "2021-10-16T16:00:01.995Z"
},
{
"ExtractedFileId": "VjF8fDkwMjQzODM5NA",
"ReportExtractionId": "548722918",
"ScheduleId": "0x07b1b9370a7d1892",
"FileType": "Note",
"ExtractedFileName": "DailyOutput20211016-170001.csv.notes.txt",
"LastWriteTimeUtc": "2021-10-16T16:00:02.000Z",
"ContentsExists": true,
"Size": 1635,
"ReceivedDateUtc": "2021-10-16T16:00:02.000Z"
}
]
}
The response contains the “ExtractedFileId” fields which are unique IDs of files. The “FileType” fields specify types of files. The “RicMaintenanceNote" file type represents a RIC Maintenance Report file, the “Full” file type represents a Full Extraction Result file, and the “Note” file type represents an Extraction Notes file.
Otherwise, you can use the following endpoints to get a file of a specific file type in a report extraction.
· Use the following endpoint with the HTTP GET method to get a RIC maintenance report file for a report extraction.
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ReportExtractions('548722918')/RicMaintenanceFile
'548722918' is the ReportExtractionId retrieved from the previous step.
The response contains a RIC maintenance report file for that report extraction.
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ExtractedFiles/$entity",
"ExtractedFileId": "VjF8fDkwMjQzODM5Ng",
"ReportExtractionId": "548722918",
"ScheduleId": "0x07b1b9370a7d1892",
"FileType": "RicMaintenanceNote",
"ExtractedFileName": "DailyOutput20211016-170001.ric.csv",
"LastWriteTimeUtc": "2021-10-16T16:00:01.998Z",
"ContentsExists": true,
"Size": 90,
"ReceivedDateUtc": "2021-10-16T16:00:01.998Z"
}
· Use the following endpoint with the HTTP GET method to get an extraction notes file for a report extraction.
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ReportExtractions('548722918')/NotesFile
'548722918' is the ReportExtractionId retrieved from the previous step.
The response contains an extraction notes file for that report extraction.
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ExtractedFiles/$entity",
"ExtractedFileId": "VjF8fDkwMjQzODM5NA",
"ReportExtractionId": "548722918",
"ScheduleId": "0x07b1b9370a7d1892",
"FileType": "Note",
"ExtractedFileName": "DailyOutput20211016-170001.csv.notes.txt",
"LastWriteTimeUtc": "2021-10-16T16:00:02.000Z",
"ContentsExists": true,
"Size": 1635,
"ReceivedDateUtc": "2021-10-16T16:00:02.000Z"
}
· Use the following endpoint with the HTTP GET method to get a full extraction result file for a report extraction.
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ReportExtractions('548722918')/FullFile
'548722918' is the ReportExtractionId retrieved from the previous step.
The response contains a full extraction result file for that report extraction.
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#ExtractedFiles/$entity",
"ExtractedFileId": "VjF8fDkwMjQzODM5NQ",
"ReportExtractionId": "548722918",
"ScheduleId": "0x07b1b9370a7d1892",
"FileType": "Full",
"ExtractedFileName": "DailyOutput20211016-170001.csv",
"LastWriteTimeUtc": "2021-10-16T16:00:01.995Z",
"ContentsExists": true,
"Size": 708,
"ReceivedDateUtc": "2021-10-16T16:00:01.995Z"
}
The ExtractedFileId will be used in the next step to download a file.
4. Download a file
The last step is downloading a file from the retrieved ExtractedFileId. The following endpoint can be used with the HTTP GET method to download a file.
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractedFiles('VjF8fDkwMjQzODM5NQ')/$value
'VjF8fDkwMjQzODM5NQ' is the ExtractedFileId of the full extraction result file retrieved from the previous step.
The response contains content in the file.
CUSIP,ISIN,SEDOL,Asset SubType,Asset Status Description,Asset SubType,Asset SubType Description,Conversion Ratio,Econometric Corp Modified Duration,EOM Worst Corp Modified Duration,Maturity Corp Modified Duration,Security Description,Shares Amount,Shares Amount Change Date,Shares Amount Type,Shares Amount Type Description,Worst Corp Modified Duration
,TH0001010006,6077008,ODSH,Issued,ODSH,Ordinary shares,,,,,BANGKOK BANK ORD,"1,908,842,894",12/31/2012,LIS,Listed,
,JP3634600005,6900546,ODSH,Issued,ODSH,Ordinary shares,,,,,TOYOTA INDUSTRI ORD,"325,840,640",06/30/2021,ISS,Issued,
,KYG217651051,BW9P816,ODSH,Issued,ODSH,Ordinary shares,,,,,CKH HOLDINGS ORD,"3,836,528,000",10/11/2021,OUT,Outstanding,
Examples
There are several examples available on GitHub which demonstrate how to use DSS REST API to download the last completed extraction files by a schedule name.
Summary
Users can use DSS REST API to create schedule extractions or on-demand extractions to extract data from DSS. Scheduled Extractions occur at a pre-defined moment in time while OnDemand extractions occur immediately when requested. This article explains the steps to use the DSS REST API to download extraction files created by scheduled extractions.
There are four steps to download extraction files created by a schedule name via the DSS REST API.
1. Get a ScheduleId from a schedule name
2. Get completed report extractions from a ScheduleId
3. Get a list of files in a report extraction
4. Download a file
Users can also refer to the examples on GitHub which demonstrate how to use DSS REST API to download the last completed extraction files by a schedule name.
