This article describes Price availability of End of Day Pricing report template. It demonstrates how to extract End of Day data using Data Availability trigger and extract data at the estimated data availability time.
Prices availability in End of Day Pricing report template
EOD Pricing reports provide initial and validated end of day prices for all exchange‐traded instruments from the real‐time network. Non‐exchange‐traded instruments are sourced from LSEG Evaluated Pricing Service, as well as from various contributors.
By default, it retrieves the most recent validated prices or analytics regardless of when instruments were priced. The End of Day prices are generated at a time after market closed. If the instrument list contains instruments which are on different region, it is possible that results are mix up of today’s and available previous day’s prices.
For example, there are 3 RICs in instrument list. Each RICs are in difference time zones; Tokyo Stock Exchange, London Stock Exchange and New York Stock Exchange. If you extract data at 17/03/2020 09:00 GMT (only TSE is closed), only TSE’s RIC will provide today’s price while other RIC will provide previous day’s price.
| RIC | Bid Price | Ask Price | Trade Date |
| BBL.BK | 95.5 | 95.75 | 16/03/2020 |
| 6201.T | 4580 | 4595 | 17/03/2020 |
| 0001.HK | 53.45 | 53.5 | 17/03/2020 |
EOD Pricing report has options to specify the type of pricing or analytics to retrieve.
- Last Updated (Default)
Retrieve the most recent validated prices or analytics, regardless of when the instruments were priced (default). The previous trading day’s validated prices are retrieved, if available; otherwise, null values can be output instead for Fixed Income, Equity, Derivatives, Money and Fund.
- Today Only
Instruments with prices or analytics from dates prior to today will be suppressed and identified in your extraction notes file. Today is calculated using the time zone set in your Preferences. For example, if your time zone is set to Eastern Time (US & Canada), then today is considered the time period from 12:00 AM to 11:59 PM EST/EDT on today's date.
For a list of fields subject to Today Only suppression, see the LSEG DataScope Select Data Content Guide. The Today Only option will identify the fields subject to suppression and the price date used in the comparison with the current date.
This option is also available on On Demand Extraction REST API via “LimitReportToTodaysData” condition. Below is the sample.
Request:
{
"ExtractionRequest":
{
"@odata.type":
"#DataScope.Select.Api.Extractions.ExtractionRequests.EndOfDayPricingExtractionRequest",
"ContentFieldNames": [
"Bid Price",
"Ask Price",
"Trade Date"
],
"IdentifierList": {
"@odata.type":
"#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers": [
{ "Identifier": "BBL.BK", "IdentifierType": "Ric" },
{ "Identifier": "6201.T", "IdentifierType": "Ric" },
{ "Identifier": "0001.HK", "IdentifierType": "Ric" }
]
},
"Condition": {
"LimitReportToTodaysData": true
}
}
}
Response:
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#DataScope.Select.Api.Extractions.ExtractionRequests.ExtractionResult",
"Contents": [
{
"IdentifierType": "Ric",
"Identifier": "BBL.BK",
"Bid Price": null,
"Ask Price": null,
"Trade Date": "2020-03-16"
},
{
"IdentifierType": "Ric",
"Identifier": "6201.T",
"Bid Price": 4580,
"Ask Price": 4595,
"Trade Date": "2020-03-17"
},
{
"IdentifierType": "Ric",
"Identifier": "0001.HK",
"Bid Price": null,
"Ask Price": null,
"Trade Date": "2020-03-16"
}
],
"Notes": [
"...S Columns for (RIC,BBL.BK,SET) suppressed due to trade date other than today
Columns for (RIC,0001.HK,HKG) suppressed due to trade date other than today..."
What if, you want the extraction to retrieve only available today’s prices?
DataScope Select provides “Data Availability” trigger option for the Schedule Extraction. With this trigger, the extraction will execute upon the release of today’s prices or analytics for the instruments in your instrument list. DataScope Select automatically detects the type of instruments in your instrument list and waits for the release of those prices or analytics to initiate the extraction. This option is applicable for Composite, EOD Pricing (excluding EOD Pricing extractions that use Criteria Lists as input), Premium EOD Pricing, Premium Pricing and Technical Indicators extractions.
The option can be selected for the report template listed above when schedule is created.
Once the schedule is created, it provides data availability times for the extraction. The data availability times are pre-calculated once the instrument list is attached to the schedule.
You can find more details of triggers using “View Triggers” button. It contains the information regarding data availability time of each instrument. “Status” - status of data availability, “Average Arrival” – average arrival time.
The schedule will be executed once today price of all instrument is arrived. To ensure optimal delivery of the current day’s prices or analytics, make sure your instruments are all regionally based (from the same time zone) and are the same asset type.
This functionality is also available via REST API. Below is the sample of REST API requests. For other steps such as Creating Instrument List, Report template and Extract result, please see the Tutorial 12: GUI control calls: immediate extract.
- Create Schedule Extraction with Data Availability Trigger
Method: POST
URL: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/Schedules
ฺRequest Body:
{
"Name": "dataAvailabilitySchedule",
"TimeZone": " Coordinated Universal Time"
"Recurrence": {
"@odata.type": "#DataScope.Select.Api.Extractions.Schedules.SingleRecurrence",
"ExtractionDateTime": "2020-03-05T00:00:00.000Z",
"IsImmediate": false
},
"Trigger": {
"@odata.type": "#DataScope.Select.Api.Extractions.Schedules.DataAvailabilityTrigger",
"LimitReportToTodaysData": true,
},
"ListId": "0x0580701d9d5b1f86",
"ReportTemplateId": "0x0580701dcd1b1f86"
}
- Get Instrument Trigger Details
Method: GET
URL: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ScheduleGetInstrumentTriggerDetails(Id='<schedule Id> ')
Response:
{
"@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#Collection(DataScope.Select.Api.Extractions.Schedules.InstrumentTriggerDetail)",
"value": [
{
"AverageArrivalUtc": "2020-03-06T10:22:12.000Z",
"DataDateUtc": "2020-03-05T00:00:00Z",
"Description": "BANGKOK BANK ORD",
"Id": "121145853701559",
"InstrumentId": "BBL.BK",
"LastArrivalUtc": "2020-03-05T10:22:37.000Z",
"Source": "SET",
"Status": "Waiting",
"Trigger": "IPC - DSE_0017P"
},
{
"AverageArrivalUtc": "2020-03-07T06:21:11.000Z",
"DataDateUtc": "2020-03-06T00:00:00Z",
"Description": "TOYOTA INDUSTRI ORD",
"Id": "121145853701560",
"InstrumentId": "6201.T",
"LastArrivalUtc": "2020-03-06T06:21:43.000Z",
"Source": "TYO",
"Status": "Arrived",
"Trigger": "IPC - DSE_0002P"
},
{
"AverageArrivalUtc": "2020-03-07T08:36:36.000Z",
"DataDateUtc": "2020-03-06T00:00:00Z",
"Description": "CKH HOLDINGS ORD",
"Id": "121145853701561",
"InstrumentId": "0001.HK",
"LastArrivalUtc": "2020-03-06T08:36:01.000Z",
"Source": "HKG",
"Status": "Arrived",
"Trigger": "IPC - DSE_0001P"
}
]
}
Application Implementation Overview
I have created a Python application demonstrating this request. It also determines overall Average Arrival time for all instruments in the trigger details, and then use Advance Python Scheduler library to schedule a job to attempt data extraction at the overall arrival time.
Overview steps:
- Login to DSS to get token
- Create Instrument List
- Append instruments to the list
- Create Schedule with data availability trigger
- Get Instrument Trigger details for the created Schedule
- Create Schedule job
- After the schedule time, Poll Extraction status
- Retrieve the data from server
- Cleanup by deleting Instrument List, Report Template and Schedule.
Almost steps are described in the Tutorial 12: GUI control calls: immediate extract. The differences are regarding getting Trigger details and scheduling a job to execute data retrieval.
- Getting Trigger details
Application calls the GetInstrumentTriggerDetails endpoint by passing the Schedule ID. Application then iterates through each trigger details to determine which trigger has last AverageArrivalUtc.
for itemInfo in m:
if itemInfo['Status'] != 'Arrived':
#"AverageArrivalUtc": "2020-02-18T06:20:59.000Z"
arrivalTime = datetime.datetime.strptime(itemInfo['AverageArrivalUtc'] , '%Y-%m-%dT%H:%M:%S.000Z')
if overallArrivalTime is None:
overallArrivalTime = arrivalTime
elif arrivalTime > overallArrivalTime:
overallArrivalTime = arrivalTime
print ("Overall AverageArrivalUtc for all instruments is " + str(overallArrivalTime) + "(UTC)")
return overallArrivalTime
Scheduling a job to execute data retrieval
Next, application utilizes Advance Python Scheduler Library (apscheduler) to create a scheduled job. The job will execute pollForExtraction function to poll Extraction status and get extracted data. After that application stays idle until the schedule is executed.
overallAverageArrivalUtc = getInstrumentTriggerDetail(scheduleId)
if overallAverageArrivalUtc is None:
# All trigger's status is Arrived, attempt to start polling for extraction data.
pollForExtraction(scheduleId)
cleanUp()
else:
# Some triggers are not arrived, then create background schedule.
scheduler = BackgroundScheduler()
scheduler.timezone= pytz.utc
scheduler.add_job(pollForExtraction, 'date', run_date=overallAverageArrivalUtc, args=[scheduleId])
scheduler.start()
try:
# This is here to simulate application activity (which keeps the main thread alive).
while (done==False):
time.sleep(5)
cleanUp()
except (KeyboardInterrupt, SystemExit):
# Not strictly necessary if daemonic mode is enabled but should be done if possible
scheduler.shutdown()
cleanUp()
Preparation steps to run the application
The application can be downloaded from Github.
- Configure DSS username and password
Login = '<username>'
Password = '<password>'
- Configure RIC list. Please note that the RICs should be in the same regions for optimization purpose.
# STEP 2 create Instrument List
listId = createInstrumentList()
appendInstrument(listId,["BBL.BK","6201.T","0001.HK"])
- Please ensure that there is no Instrument Name: “myInstrumentList”, Report Template Name: “myEodTemplateName” and Schedule Name: “dataAvailabilitySchedule” created in your DSS account.
Output console
INFO, Successfully created token
INFO, 2020-03-17-16:32:40, Create Instrument List for myInstrumentList sucessfully where ListId = 0x0705a7590c716ee2
INFO, 2020-03-17-16:32:43, Append Instruments successfully and Appended Instrument Count = 3
INFO, 2020-03-17-16:33:00, Created Report Templates Id: 0x0705a7178c616ee6
INFO, 2020-03-17-16:33:02, Created Schedule Id: 0x0705a852a7e16ee8
[
{
"AverageArrivalUtc": "2020-03-17T10:22:28.000Z",
"DataDateUtc": "2020-03-16T00:00:00Z",
"Description": "BANGKOK BANK ORD",
"Id": "121146813732602",
"InstrumentId": "BBL.BK",
"LastArrivalUtc": "2020-03-16T10:22:49.000Z",
"Source": "SET",
"Status": "Waiting",
"Trigger": "IPC - DSE_0017P"
},
{
"AverageArrivalUtc": "2020-03-18T06:22:13.000Z",
"DataDateUtc": "2020-03-17T00:00:00Z",
"Description": "TOYOTA INDUSTRI ORD",
"Id": "121146813732603",
"InstrumentId": "6201.T",
"LastArrivalUtc": "2020-03-17T06:21:59.000Z",
"Source": "TYO",
"Status": "Arrived",
"Trigger": "IPC - DSE_0002P"
},
{
"AverageArrivalUtc": "2020-03-18T08:36:34.000Z",
"DataDateUtc": "2020-03-17T00:00:00Z",
"Description": "CKH HOLDINGS ORD",
"Id": "121146813732604",
"InstrumentId": "0001.HK",
"LastArrivalUtc": "2020-03-17T08:36:34.000Z",
"Source": "HKG",
"Status": "Arrived",
"Trigger": "IPC - DSE_0001P"
}
]
Overall AverageArrivalUtc for all instruments is 2020-03-17 10:22:28(UTC)
