Introduction
Working with Refinitiv real-time data, you probably already or is going to come across a specific type of instrument called Chains. The concept of chains is relatively straightforward and decoding them is not that complicated.
In this article, I'm going to introduce you to a simple and easy way to retrieve the current constituents of a Chain RIC by using an endpoint provided in Refinitiv Data Platform APIs. However, to understand more about a Chain RIC and its structure, we're going to use another way, which is using the Refinitiv Real-time market data first. There are many APIs that can be used to retrieve these data, you may check our Real-time API Catalog for more detail. The Refinitiv WebSocket API is used to retrieve the real-time data here as an example of how to retrieve RICs and their data, and then process them.
What are Chains?
Chains are used to hold instrument names that have a common association. For example, a chain may contain the constituents of a market index like the Dow Jones or the list of strike prices on a particular option contract. It is important to note that chains contain the instrument names (a.k.a. RICs - Refinitiv Instrument Codes) but not the values of the instruments. If you are interested in the instruments’ values, you need to first subscribe to the chain to get the names of the constituent, then subscribed to these constituents to get their values.
The chain data structure
To start working with the chain using the WebSocket API below is the diagram representing the chain structure and some details that are good to know. (However, RDP API is already covering this part for you! as mentioned, it's simple and easy) The chain itself is represented by the boxes with a black header, and the boxes with a grey header represent the constituents referenced by the chain and their value
From this diagram. in this particular example, the chain is composed of 8 instruments (the 8 boxes with black header) called Chain Records.
These Chain Records are linked together and constitute the complete chain. A Chain record is a MarketPrice instrument that represents a piece of a chain. Each Chain Record contains the name of up to 14 chain elements. A complete chain spans over one or several Chain Records that are linked together thanks to dedicated fields. These fields enable users and applications to navigate through the chain.
In our FTSE 100 index example above, we could see that:
- The chain is made of 8 Chain Records named 0#.FTSE, 1#.FTSE, 2#.FTSE, ..., 7#.FTSE
- These instruments are linked together via their LONGPREVLR and LONGNEXTLR fields (blue lines in the diagram)
- The LONGPREVLR field of 0#.FTSE is empty, indicating it’s the first Chain Record of the chain. The LONGNEXTLR of the 7#.FTSE is also empty, indicating it is the last Chain Record.
- The chain links to 102 elements (RICs). The first two elements (.FTSE: FTSE 100 INDEX and .AD.FTSE: London Stock Exchange MKT Stats) are the summary RICs of the FTSE index. The other 100 elements are the RICs of the constituents of the FTSE.
- As the 14 LONGLINKs of the first seven Chain Records are used, their REF_COUNT fields are set to 14. As the last Chain Record only holds 4 elements, its REF_COUNT field is set to 4 meaning that only the first four LONGLINKS fields are used.
- Chains only contain the names of their elements, not their values. To get the element's values, you have to subscribe to these elements separately.
- If the application is interested in the element’s values, it must subscribe to these elements separately. In our FTSE example, if you’re interested in the values of the index, you must send a subscription request for .FTSE If you are interested in the values of one of its constituents – let’s say Airtel Africa PLC – you must send a subscription request for AAF.L In the diagram above 3 of the FTSE constituents (Airtel Africa PLC, B&M European Value Retail SA, and Whitbread PLC) and their values are represented by the boxes with a grey header.
- There are 3 types of record templates that Chain Records may use. They provide the same functionalities but use different fields to hold equivalent values due to each template supporting different RIC sizes. In this article with the FTSE index, we're using Template #85. The other 2 types of record templates are explained in this article. If you intend to build an application that is able to work with any chains available on the real-time platform, it's recommended to support all the 3 templates. On the other hand, if you know that your application will only work with a limited set of well-identified chains, then you can reasonably limit your implementation to the templates used by these chains (like in this article - we're retrieving the data of the FTSE index that is on Template #85).
- For more detail regarding the Chain, these articles Simple Chain objects for EMA below explain the chain concept and its structure in detail
Since we already know the basic structure of the Chain, we're ready to retrieve chain constituents. Before using the simple and easy endpoint provided in RDP API, let me show you the first method to retrieve data from the Real-time platform, with Websocket API!
To retrieve data in both ways, the OAuth access token is required. Hence the utility file rdpToken.py provided in the GitHub repository is used to check if the token is valid and generate/refresh tokens if needed. (The file and other examples to work with RDP API are provided in Tutorials source code: Quickstart and Python tutorials source code, Python starter examples, RDP API - Authorization in Python. You may check RDP API General Guidelines before working with RDP API)
** These are meant to be examples for illustration purposes only and not production-ready code. Features such as item recovery, connection recovery, etc., must be built into the application.
1) Get chain constituents with Refinitiv Websocket API
a WebSocket connection can be made to connect with Refinitiv Real-Time Distribution Systems, RTDS (version 3.2.1 (and higher) where this transport is enabled), and, Refinitiv Real-Time - Optimized, RTO (cloud solution) available via Refinitiv Data Platform (RDP). In this article, we are going to make a connection to the latter one (RTO) which is highlighted in the green box below
It's recommended to check the Refinitiv WebSocket API Catalog page,its quick start guide, and its tutorial: Connect to Refinitiv Real-Time - Optimized. Plus, this GitHub repository contains example programs used to illustrate an implementation of the protocol to make WebSocket connections to RTDS and to RTO.
To retrieve the data, first, we are going to do the authentication via RDP to get an access token required for the connection to a Refinitiv Real-Time service available in RTO. Then, Discover which RTO endpoint to connect to by making a service discovery request to RDP and using this information to connect to RTO to receive Real-Time content. Let's go through the Python code snippets below with an explanation of each step.
First, import necessary Python libraries (including the rdpToken.py used to manage an access token) with assign the necessary constants, such as the service discovery endpoint URL, then put the chain RIC you're interested in to the variable base_ric
Note: In this article, we're making a non-streaming mode request to RTO (snapshot = True), as we only want to retrieve a snapshot of the chain's constituent list. The Chain's constituent list is almost static and won't be changed very often. However, the list can be changed, for more detail, check this article regarding chain updates
import rdpToken
import requests, json, time, getopt, configparser
from datetime import datetime, timedelta
import websocket, socket
import threading
from threading import Thread, Event
import time
# Websocket - RTO Constants
service_discovery_url = 'https://api.refinitiv.com/streaming/pricing/v1/?transport=websocket'
hostname = ''
port = ''
app_id = '123'
auth_token = ''
position = socket.gethostbyname(socket.gethostname())
snapshot = True # True for non-streaming request
base_ric = '0#.FTSE' # chain RIC used to retrieve the data
all_underlying_rics = [] # to store the result
# other application variable
retrieved_all_rics = False
web_socket_app = None
web_socket_open = False
error_occurred = False
In the main part of the application, request an access token from RDP API (with functions in rdpToken), then uses the access token to discover RTO host/port with function set_rto_host_port.
Then start the WebSocket connection by starting a WebSocket handshake, send the login request (with access token) once getting a callback message when WebSocket is open, and create a thread for WebSocket application processing. Which is working as a diagram below.
The Jupyter notebook file of the full code can be found here
- The application is going through each Chain Record, then set the current LONGNEXTLR as the next Chain Record to send a request to the server
- When LONGNEXTLR is empty, set the flag retrieved_all_rics = True to tell the application that all the constituent RICs are retrieved
The code used for the message processing is implemented in the function process_message as below
def process_message(ws, message_json):
global auth_token, retrieved_all_rics, error_occurred
""" Parse at high level and output JSON of message"""
message_type = message_json['Type']
# receive refresh message
if message_type == "Refresh":
# process login response
if 'Domain' in message_json:
print('Process Login message')
message_domain = message_json['Domain']
if message_domain == "Login":
process_login_response(ws, message_json)
# process underlying RICs response
elif 'Fields' in message_json:
print('Process Fields message (Underlying RICs)')
message_fields = message_json['Fields']
for x in range(1,15):
current_field = 'LONGLINK{}'.format(x)
value = message_fields[current_field]
if value is not None:
all_underlying_rics.append(value)
longnextlr = message_fields['LONGNEXTLR']
# if reached the last chain record, set retrieved_all_rics flag
if longnextlr is None:
print('All underlying RICs are retrieved')
retrieved_all_rics = True
# if not reached the last chain record yet, request longnextlr as a next chain record
else:
send_chain_record_element_request(ws, longnextlr)
elif message_type == "Status":
print('Get Status message: ' + str(message_json['State']))
error_occurred = True
# receive ping message
elif message_type == "Ping":
pong_json = { 'Type':'Pong' }
ws.send(json.dumps(pong_json))
print("SENT: Pong")
Lastly, keep checking if all the constituent RICs are retrieved, and close the WebSocket application. Then print all RICs retrieved. The main code is as below
if __name__ == "__main__":
auth_token = rdpToken.getToken()
set_rto_host_port(auth_token)
# Start websocket handshake
ws_address = "wss://{}:{}/WebSocket".format(hostname, port)
print("Connecting to WebSocket " + ws_address + " ...")
web_socket_app = websocket.WebSocketApp(ws_address, header=['User-Agent: Python'],
on_message=on_message,
on_error=on_error,
on_close=on_close,
subprotocols=['tr_json2'])
# Callback for once websocket is open - which will send the Login request
web_socket_app.on_open = on_open
# Create Thread for WebsocketApp processing
wst = threading.Thread(target=web_socket_app.run_forever)
wst.start()
try:
while True:
if(retrieved_all_rics):
web_socket_app.close() # Close websocket app once all RICs are retrieved
break
except KeyboardInterrupt:
pass
finally:
print(str(len(all_underlying_rics)) + ' underlying RICs are retrieved, which are')
print(all_underlying_rics)
Here is the console output of the application, at the end, all constituents RICs (underlying RICs) are printed as Python list)
Existing token read from: token.txt
Token expired, refreshing a new one...
Read credentials from file
Saving the new token
Retrieved RTO host: xxxx-api.refinitiv.net:443
Connecting to WebSocket wss://xxxx-api.refinitiv.net:443/WebSocket ...
WebSocket successfully connected!
RECEIVED: message
Process Login message
SENT request of RIC:0#.FTSE
RECEIVED: message
Process Fields message (Underlying RICs)
SENT request of RIC:1#.FTSE
RECEIVED: message
Process Fields message (Underlying RICs)
SENT request of RIC:2#.FTSE
RECEIVED: message
Process Fields message (Underlying RICs)
SENT request of RIC:3#.FTSE
RECEIVED: message
Process Fields message (Underlying RICs)
SENT request of RIC:4#.FTSE
RECEIVED: message
Process Fields message (Underlying RICs)
SENT request of RIC:5#.FTSE
RECEIVED: message
Process Fields message (Underlying RICs)
SENT request of RIC:6#.FTSE
RECEIVED: message
Process Fields message (Underlying RICs)
SENT request of RIC:7#.FTSE
RECEIVED: message
Process Fields message (Underlying RICs)
All underlying RICs are retrieved
WebSocket Closed
102 underlying RICs are retrieved, which are
['.FTSE', '.AD.FTSE', 'AAF.L', 'AAL.L', 'ABDN.L', 'ABF.L', 'ADML.L', 'AHT.L', 'ANTO.L', 'AUTOA.L', 'AV.L', 'AVST.L', 'AVV.L', 'AZN.L', 'BAES.L', 'BARC.L', 'BATS.L', 'BDEV.L', 'BKGH.L', 'BLND.L', 'BMEB.L', 'BNZL.L', 'BP.L', 'BRBY.L', 'BT.L', 'CCH.L', 'CNA.L', 'CPG.L', 'CRDA.L', 'CRH.L', 'DCC.L', 'DGE.L', 'DPH.L', 'EDV.L', 'ENT.L', 'EXPN.L', 'FLTRF.L', 'FRES.L', 'GLEN.L', 'GSK.L', 'HIK.L', 'HLMA.L', 'HLN.L', 'HRGV.L', 'HSBA.L', 'HWDN.L', 'ICAG.L', 'ICP.L', 'IHG.L', 'III.L', 'IMB.L', 'INF.L', 'ITRK.L', 'JD.L', 'KGF.L', 'LAND.L', 'LGEN.L', 'LLOY.L', 'LSEG.L', 'MGGT.L', 'MNDI.L', 'MNG.L', 'MRON.L', 'NG.L', 'NWG.L', 'NXT.L', 'OCDO.L', 'PHNX.L', 'PRU.L', 'PSHP.L', 'PSN.L', 'PSON.L', 'REL.L', 'RIO.L', 'RKT.L', 'RMV.L', 'RR.L', 'RS1R.L', 'RTO.L', 'SBRY.L', 'SDR.L', 'SGE.L', 'SGRO.L', 'SHEL.L', 'SJP.L', 'SKG.L', 'SMDS.L', 'SMIN.L', 'SMT.L', 'SN.L', 'SPX.L', 'SSE.L', 'STAN.L', 'SVT.L', 'TSCO.L', 'TW.L', 'ULVR.L', 'UTG.L', 'UU.L', 'VOD.L', 'WPP.L', 'WTB.L']
2) Get chain constituents with Refinitiv Data Platform API
Here comes an easier and simpler way! with only one endpoint in RDP API, we can retrieve constituent lists of Chain RIC in one call (not counting a call to retrieve access token though). The example of this API call for this endpoint can be found in the API Playground: /data/pricing/chains/v1/. For more detail about token, please check RDP API - Authorization - All about tokens
With this method, we can retrieve the chain constituents list with just one API call instead of looping through each Chain Record and sending requests to the server multiple times. We also don't need to use WebSocket to make a connection to the Real-time platform so no need to implement the message handler functions. Lastly, we don't have to know the chain structure and handle each chain's record template because this endpoint is already handling it for us!
Just in case the error occurs or the endpoint cannot be called properly, the code to handle unexpected results is implemented as below.
The Jupyter notebook file of the full code can be found here
import rdpToken
import http.client
import json
import time
def get_underlying_rics_rdp(base_ric):
try:
auth_token = rdpToken.getToken()
print(f'Retrieving underlying RICs of chain RIC: ', base_ric)
conn = http.client.HTTPSConnection("api.refinitiv.com")
payload = ''
headers = {
'Authorization': 'Bearer ' + auth_token
}
conn.request('GET', f'/data/pricing/chains/v1/?universe={base_ric}&limit=100', payload, headers)
r = conn.getresponse()
data = r.read()
data = json.loads(data.decode("utf-8"))
except requests.exceptions.RequestException as e:
print('Refinitiv Data Platform authentication exception failure:', e)
if r.status == 200:
print('Underlying RICs of chain are successfully retrieved')
elif r.status in [400, 401, 403, 404, 410, 451, 503]:
print('Error retrieving underlying RICs (to be retried):', r.status, r.reason)
time.sleep(5)
return get_underlying_rics_rdp()
elif data == b'':
print('Error retrieving underlying RICs (to be retried): get empty response')
time.sleep(5)
return get_underlying_rics_rdp()
else:
# Retry the request to the API gateway
print('Retrieval of underlying RICs of chain HTTP code:', r.status, r.reason)
time.sleep(5)
print('Retry change request')
return get_underlying_rics_rdp()
return data
if __name__ == "__main__":
base_ric = '0#.FTSE'
data = get_underlying_rics_rdp(base_ric)
print(str(len(data['data']['constituents'])) + ' underlying RICs are retrieved, which are')
print(data['data']['constituents'])
Here is the console output of the application, at the end, all constituents RICs (underlying RICs) are printed as Python list)
Existing token read from: token.txt
Retrieving underlying RICs of chain RIC: 0#.FTSE
Underlying RICs of chain are successfully retrieved
102 underlying RICs are retrieved, which are
['.FTSE', '.AD.FTSE', 'AAF.L', 'AAL.L', 'ABDN.L', 'ABF.L', 'ADML.L', 'AHT.L', 'ANTO.L', 'AUTOA.L', 'AV.L', 'AVST.L', 'AVV.L', 'AZN.L', 'BAES.L', 'BARC.L', 'BATS.L', 'BDEV.L', 'BKGH.L', 'BLND.L', 'BMEB.L', 'BNZL.L', 'BP.L', 'BRBY.L', 'BT.L', 'CCH.L', 'CNA.L', 'CPG.L', 'CRDA.L', 'CRH.L', 'DCC.L', 'DGE.L', 'DPH.L', 'EDV.L', 'ENT.L', 'EXPN.L', 'FLTRF.L', 'FRES.L', 'GLEN.L', 'GSK.L', 'HIK.L', 'HLMA.L', 'HLN.L', 'HRGV.L', 'HSBA.L', 'HWDN.L', 'ICAG.L', 'ICP.L', 'IHG.L', 'III.L', 'IMB.L', 'INF.L', 'ITRK.L', 'JD.L', 'KGF.L', 'LAND.L', 'LGEN.L', 'LLOY.L', 'LSEG.L', 'MGGT.L', 'MNDI.L', 'MNG.L', 'MRON.L', 'NG.L', 'NWG.L', 'NXT.L', 'OCDO.L', 'PHNX.L', 'PRU.L', 'PSHP.L', 'PSN.L', 'PSON.L', 'REL.L', 'RIO.L', 'RKT.L', 'RMV.L', 'RR.L', 'RS1R.L', 'RTO.L', 'SBRY.L', 'SDR.L', 'SGE.L', 'SGRO.L', 'SHEL.L', 'SJP.L', 'SKG.L', 'SMDS.L', 'SMIN.L', 'SMT.L', 'SN.L', 'SPX.L', 'SSE.L', 'STAN.L', 'SVT.L', 'TSCO.L', 'TW.L', 'ULVR.L', 'UTG.L', 'UU.L', 'VOD.L', 'WPP.L', 'WTB.L']
Conclusion
After finishing reading this article (and maybe some practice with the Python code provided), you should be able to retrieve the Chain constituents list. Two methods of retrieving the data are introduced here, please feel free to pick the one that you're comfortable with. However, you could see that using the RDP API endpoint is much simpler because the endpoint already handles the chain logic for you, unlike the Refinitiv Websocket where you have to handle the message returned from the server by yourself so you need to understand basic chain concept and its structure before working with it.
Then you can use this Chain constituents list to request the data of each constituent itself as a next step.
Last but not least, I hope you have fun with this article and if you have any questions or issues regarding API usage, please don’t hesitate to post your question on our Q&A forum
Reference
- Register or Log in to applaud this article
- Let the author know how much this article helped you
