Troubleshooting IPA & RD Python Library

Authors:

Jonathan Legrand
Developer Advocate Developer Advocate

This article is writen a little diferently to the others I normally publish. It takes on more of a Blog form. For the original article, please go to the LSEG Developer Portal.

I was working on expanding/part 2 of this article on the LSEG Developer Portal, in which I send time-series data to a service that functions, in effect, as a calculator; this service is called ‘Instrument Pricing Analytics’, IPA. Specifically: I used the LSEG Data (LD) Python Library to send index price data and wanted to retrieve back Implied Volatilities.

While doing this, I once retrieved back a data frame filled with <NA>s and I had to troubleshoot it… But where to start?

In this short article, I look into my journey troubleshooting this step, in the hope that it’s of use to you!

Background

I used certain common libraries, such as Pandas, in addition to RD library functions:

    	
            import pandas as pd
import refinitiv.data as rd # This is LSEG's Data and Analytics' API wrapper, called the Refinitiv Data Library for Python.
from refinitiv.data.content import historical_pricing # We will use this Python Class in `rd` to show the Implied Volatility data already available before our work.
import refinitiv.data.content.ipa.financial_contracts as rdf # We're going to need thtis to use the content layer of the RD library and the calculators of greeks and Impl Volat in Instrument Pricing Analytics (IPA) and Exchange Traded Instruments (ETI)
from refinitiv.data.content.ipa.financial_contracts import option # We're going to need thtis to use the content layer of the RD library and the calculators of greeks and Impl Volat in IPA & ETI

# Let's authenticate ourseves to LSEG's Data and Analytics service, Refinitiv:
try: # The following libraries are not available in Codebook, thus this try loop
rd.open_session(config_name="C:\\Example.DataLibrary.Python-main\\Example.DataLibrary.Python-main\\Configuration\\refinitiv-data.config.json")
rd.open_session("desktop.workspace")
except:
rd.open_session()

You can already see traces of that service I mentioned, IPA, in the 4th line. FYI: Here we are using the LD Library (named the RD Library) version 1.0.0b24.

For more information on the configuration proccess (as per line 9 in the code above), please read this quickstart.

I then gathered data on two Options, a live one (‘2HSI16200L8.HF’) and an expired one (‘HSI19300N3.HF^B23’).

    	
            HSI_test0 = rd.content.historical_pricing.summaries.Definition(
'2HSI16200L8.HF',
interval=rd.content.historical_pricing.Intervals.DAILY,
fields=['SETTLE'],
start='2023-10-01',
end='2024-01-09').get_data().data.df
hk_rf = 100 - rd.get_history(
universe=['HK3MT=RR'], # HK10YGB=EODF, HKGOV3MZ=R, HK3MT=RR
fields=['TR.MIDPRICE'],
start=HSI_test0.index[0].strftime('%Y-%m-%d'),
end=HSI_test0.index[-1].strftime('%Y-%m-%d'))
HSI_test1 = pd.merge(
HSI_test0, hk_rf, left_index=True, right_index=True)
HSI_test1 = HSI_test1.rename(
columns={"SETTLE": "OptionPrice", "Mid Price": "RfRatePrct"})
hist_HSI_undrlying_pr = rd.get_history(
universe=['.HSI'],
fields=["TRDPRC_1"],
# interval="1D",
start=HSI_test0.index[0].strftime('%Y-%m-%d'),
end=HSI_test0.index[-1].strftime('%Y-%m-%d'))
HSI_test2 = pd.merge(HSI_test1, hist_HSI_undrlying_pr,
left_index=True, right_index=True)
HSI_test2 = HSI_test2.rename(
columns={"TRDPRC_1": "UndrlyingPr"})
HSI_test2.columns.name = 'STXE42000D3.EX'
HSI_test2

Content Layer

There happen to be three layers to the Data Library. The 1st, called the Access Layer, is the most user-friendly; the 2nd, the Content Layer, is less so, and the Delivery Layer simply sends data requests to LSEG Databases and returns datasets in their ‘rawest’ form — with no/few filters. Below, we will use the Content Layer:

    	
            request_fields = ['MarketValueInDealCcy', 'RiskFreeRatePercent', 'UnderlyingPrice', 'PricingModelType', 'DividendType', 'UnderlyingTimeStamp', 'ReportCcy', 'VolatilityType', 'Volatility', 'DeltaPercent', 'GammaPercent', 'RhoPercent', 'ThetaPercent', 'VegaPercent', 'ErrorMessage']
        
        
    
    	
            response = option.Definition(
buy_sell=option.BuySell.BUY,
call_put=option.CallPut.CALL,
instrument_code=live_inst,
strike=float(4200),
underlying_definition=option.EtiUnderlyingDefinition(instrument_code=live_inst),
underlying_type=option.UnderlyingType.ETI,
fields=request_fields,
pricing_parameters=option.PricingParameters(
pricing_model_type='BlackScholes',
volatility_type="Implied",
market_value_in_deal_ccy=float(HSI_test2['OptionPrice'][0]),
report_ccy="HKD",
risk_free_rate_percent=float(HSI_test2['RfRatePrct'][0]),
underlying_price=float(HSI_test2['UndrlyingPr'][0]),
valuation_date=HSI_test0.index[0].strftime('%Y-%m-%d')
)).get_data()

I used VSCode and Intellisense to figure out which arguments were needed in `option.Definition`:

Intellisense is great to quickly write functions like this, however, as we will see soon, it’s not enough.

My code did not work… I kept getting errors:

How can I investigate such an issue?

What this Content Layer does is send request to LSEG’s back-end to a specific endpoint. One can see these endpoints on the API Playground:

 

How can you find the endpoint in question that you used? this is where ‘Logs’ come in:

Switching `Logs` On

You can switch on logs with the following code in Python:

    	
            rd.get_config().set_param(
param=f"logs.transports.console.enabled", value=True
)
session = rd.open_session()
session.set_log_level("DEBUG")

You can also switch on logs with the configuration file. If we look into the configuration file used to authenticate ourselves to RD, we can change the “logs”>“transports” >”file”>”enabled” to true, giving us a log file in the path where we are running the Python code. The one produced from this on my machine was called 20240109–1353–11136-refinitiv-data-lib.log. It started with

[2024-01-10T10:59:21.083581+01:00] - [sessions.desktop.workspace.0] - [DEBUG] - [4772 - MainThread] - [_session_provider] - [session_provider] -  + Session created: DesktopSession

name = 'workspace'

connection = DesktopConnection

stream_auto_reconnection = True

handshake_url = http://localhos:9000/api/handshake

state = OpenState.Closed

session_id = 0

logger_name = sessions.desktop.workspace.0

You can check which session you are using, out of the Desktop and Platform session, on lign 8. More on these sessions here.

From there, I was looking for the JSON message sent to the IPA service. To find those, I looked for the last instance of api/rdp/data, since we’re looking into our last usage of the RDP API in our code. I looked for the last instance of this and found the request which started with the following:

url = http://localhos:9001/api/rdp/data/quantitative-analytics/v1/financial-contracts
method = POST
headers = {'Content-Type': 'application/json', ... }
json = {'universe': [{'instrumentType': 'Option', 'instrumentDefinition': ... 'ErrorMessage']}
[2024-01-10T10:59:21.083588+01:00] - [RetryTransportBase] - [DEBUG] - [13552 - MainThread] - [_retry_transport] - [_handle_request] - Sending request to http://localhos:9001/api/rdp/data/quantitative-analytics/v1/financial-contracts
[2024-01-10T10:59:23.670464+01:00] - [sessions.desktop.workspace.0] - [DEBUG] - [13552 - MainThread] - [http_service] - [request] - HTTP Response id 9
status_code = 200
text = {"headers":[{"type":"Float","name":"Strike"}, ... , {"type":"String","name":"ErrorMessage"}],"data":[[4200.0,4804.0,1.146,17195.84,"BlackScholes", ... ,"NaN","NaN","NaN",,"NaN","Unable to calculate the Implied Volatility."]]}

It is too long to show in full, but in line 1 & 5, you can clearly see the endpoint that was pinged. Here it’s `quantitative-analytics/v1/financial-contracts`.

API Playground

Coming back to the API Playground, we can now find all the documentation we’re after:

Lookinginto the examples given in the ‘Playground’ tab, I managed to construst a ‘body’ that gave some data similar to what I was after by ommiting data that I originally thought was useful. I could see the data returned by pressing on ‘SEND’:

 

The body that seem to give me some data was:

{
"instrumentType": "Option",
"instrumentDefinition": {
"buySell": "Buy",
"underlyingType": "Eti",
"instrumentCode": 'STXE42000D3.EX',
},
"pricingParameters": {
"marketValueInDealCcy": 4804,
"riskFreeRatePercent": 1.1460,
"underlyingPrice": 17195.84,
"pricingModelType": "BlackScholes",
"dividendType": "ImpliedYield",
"volatilityType": "Implied",
"underlyingTimeStamp": "Default",
"reportCcy": "HKD"
}
}

But what do we do now? Well, there are two options, you could go back to the Content Layer and change the relevent arguments, or use the Delivery Layer. I haven’t shown you how to use that yet, so let’s try it:

Delivery Layer

 

    	
            

live_universe = [

        {

          "instrumentType": "Option",

          "instrumentDefinition": {

            "buySell": "buy",

            "underlyingType": "Eti",

            "instrumentCode": live_inst,

          },

          "pricingParameters": {

            "marketValueInDealCcy": float(HSI_test2['OptionPrice'][i]),

            "riskFreeRatePercent": float(HSI_test2['RfRatePrct'][i]),

            "underlyingPrice": float(HSI_test2['UndrlyingPr'][i]),

            "pricingModelType": "BlackScholes",

            "dividendType": "ImpliedYield",

            "volatilityType": "Implied",

            "underlyingTimeStamp": "Default",

            "reportCcy": "HKD"

          }

        }

      for i in range(len(HSI_test2.index))]

 

def Chunks(lst, n):

    """Yield successive n-sized chunks from lst."""

    for i in range(0, len(lst), n):

        yield lst[i:i + n]

 

batchOf = 100  # 100 is the max

for i, j in enumerate(Chunks(live_universe, 100)):

    print(f"Batch of {batchOf} requests no. {str(i+1)}/{str(len([i for i in Chunks(live_universe, batchOf)]))} started")

    # Example request with Body Parameter - Symbology Lookup

    live_troubleshoot_request_definition = rd.delivery.endpoint_request.Definition(

        method=rd.delivery.endpoint_request.RequestMethod.POST,

        url='https://api.refinitiv.com/data/quantitative-analytics/v1/financial-contracts',

        body_parameters={"fields": request_fields,

                         "outputs": ["Data", "Headers"],

                         "universe": j})

 

    live_troubleshoot_resp = live_troubleshoot_request_definition.get_data()

    headers_name = [h['name'] for h in live_troubleshoot_resp.data.raw['headers']]

 

    if i == 0:

        live_troubleshoot_df = pd.DataFrame(

            data=live_troubleshoot_resp.data.raw['data'],

            columns=headers_name)

    else:

        _live_troubleshoot_df = pd.DataFrame(

            data=live_troubleshoot_resp.data.raw['data'],

            columns=headers_name)

        live_troubleshoot_df = live_troubleshoot_df.append(_live_troubleshoot_df, ignore_index=True)

    print(f"Batch of {batchOf} requests no. {str(i+1)}/{str(len([i for i in Chunks(live_universe, batchOf)]))} ended")

Batch of 100 requests no. 1/1 started

Batch of 100 requests no. 1/1 ended

    	
            

live_troubleshoot_df.columns.name = live_inst

live_troubleshoot_df.describe(include='all')

2HSI16200L8.HF Strike MarketValueInDealCcy RiskFreeRatePercent UnderlyingPrice PricingModelType DividendType UnderlyingTimeStamp ReportCcy VolatilityType Volatility DeltaPercent GammaPercent RhoPercent ThetaPercent VegaPercent ErrorMessage
count 66 66 66 66 66 66 66 66 66 66 66 66 66 66 66 66
unique NaN NaN NaN NaN 1 1 1 1 1 NaN NaN NaN NaN NaN NaN 1
top NaN NaN NaN NaN BlackScholes ImpliedYield Default HKD Calculated NaN NaN NaN NaN NaN NaN  
freq NaN NaN NaN NaN 66 66 66 66 66 NaN NaN NaN NaN NaN NaN 66
mean 16200 4469.651515 1.144962 17133.58273 NaN NaN NaN NaN NaN 36.49445 0.549396 0.000023 246.311761 -0.60418 122.613079 NaN
std 0 616.548644 0.057065 552.082884 NaN NaN NaN NaN NaN 2.622203 0.023784 0.000003 5.615925 0.044779 1.075281 NaN
min 16200 3441 1.02 16190.02 NaN NaN NaN NaN NaN 32.32272 0.506526 0.000019 235.036064 -0.671197 120.429082 NaN
25% 16200 3804.25 1.09625 16633.4175 NaN NaN NaN NaN NaN 33.48026 0.524025 0.000021 242.250465 -0.641238 121.787159 NaN
50% 16200 4677.5 1.144 17183.985 NaN NaN NaN NaN NaN 37.94038 0.558198 0.000022 246.216584 -0.6303 122.641014 NaN
75% 16200 5004.75 1.18775 17566.2 NaN NaN NaN NaN NaN 38.56966 0.569113 0.000027 250.933266 -0.553924 123.424069 NaN
max 16200 5568 1.2895 18238.21 NaN NaN NaN NaN NaN 40.35844 0.587471 0.000029 256.999886 -0.534717 124.543976 NaN
    	
            live_troubleshoot_df
        
        
    
2HSI16200L8.HF Strike MarketValueInDealCcy RiskFreeRatePercent UnderlyingPrice PricingModelType DividendType UnderlyingTimeStamp ReportCcy VolatilityType Volatility DeltaPercent GammaPercent RhoPercent ThetaPercent VegaPercent ErrorMessage
0 16200 4804 1.146 17195.84 BlackScholes ImpliedYield Default HKD Calculated 39.00143 0.562107 0.000021 241.62978 -0.65538 121.659475  
1 16200 4772 1.138 17213.87 BlackScholes ImpliedYield Default HKD Calculated 38.67129 0.561072 0.000021 242.838419 -0.647674 121.943268  
2 16200 4974 1.095 17485.98 BlackScholes ImpliedYield Default HKD Calculated 39.15167 0.567839 0.000021 246.267948 -0.646983 122.809304  
3 16200 5016 1.094 17517.4 BlackScholes ImpliedYield Default HKD Calculated 39.35019 0.569199 0.00002 246.25132 -0.650093 122.809412  
4 16200 5235 1.1715 17664.73 BlackScholes ImpliedYield Default HKD Calculated 40.29344 0.576885 0.00002 246.282736 -0.671197 122.54165  
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
61 16200 3785 1.088 16646.41 BlackScholes ImpliedYield Default HKD Calculated 33.22061 0.523518 0.000027 244.998808 -0.54339 122.570871  
62 16200 3770 1.086 16645.98 BlackScholes ImpliedYield Default HKD Calculated 33.10405 0.522923 0.000027 245.241265 -0.540696 122.628388  
63 16200 3721 1.038 16535.33 BlackScholes ImpliedYield Default HKD Calculated 33.26945 0.519662 0.000027 242.120688 -0.542884 122.136924  
64 16200 3475 1.043 16224.45 BlackScholes ImpliedYield Default HKD Calculated 32.54961 0.507613 0.000028 236.60269 -0.534884 120.913425  
65 16200 3441 1.0895 16190.02 BlackScholes ImpliedYield Default HKD Calculated 32.32272 0.506526 0.000029 236.548565 -0.534717 120.745756  
                                 
66 rows × 16 columns                                
    	
            rd.close_session()
        
        
    

 

 

 

P.S.: Trial & Error Troubleshooting

 

I recently came uppon a second error returned from IPA, ''ErrorMessage" `Unable to calculate the Implied Volatility` and "ErrorCode" `QPS-Pricer.4011`.

I went through the technique above, and found the folowing test call totry out on the API Playground:

 

{

  "universe": [

    {

      "instrumentType": "Option",

      "instrumentDefinition": {

        "buySell": "Buy",

        "callPut": "Call",

        "underlyingType": "Eti",

        "instrumentCode": "STXE45500N4.EX",

        "strike": 4550

      },

      "pricingParameters": {

        "pricingModelType": "BlackScholes",

        "underlyingTimeStamp": "Default",

        "volatilityType": "Implied",

        "marketValueInDealCcy": 4534.61,

        "reportCcy": "EUR",

        "riskFreeRatePercent": 3.938,

        "underlyingPrice": 4534.61,

        "valuationDate": "2023-12-18T11:10:00Z"

      }

    }

  ],

  "fields": [

    "ErrorMessage",

    "ErrorCode",

    "MarketValueInDealCcy",

    "RiskFreeRatePercent",

    "UnderlyingPrice",

    "Volatility",

    "DeltaPercent",

    "GammaPercent",

    "RhoPercent",

    "ThetaPercent",

    "VegaPercent"

  ]

}

 

What I then did, on the Playground, was remove one (and only one) argument in . I found out, through this trial and error proccess that the the culprit was "marketValueInDealCcy".

Removing this argument showed that this option was trading at a much lower rate, making Greeks imposible to calculate.

 

 

References

Do not hesitate to ask any technical questions you may have about LSEG APIs on the LSEG Developer Comunity Q&A Forum. Please note, however, that this is a Forum aimed at answering technical questions onnly; for others (e.g.: content-related questions), please reach out to my.refinitiv.com.

For documentation, training articles and videos and more on LSEG APIs, please visit the LSEG Developer Portal.