Article

Use Fiddler to Capture Refinitiv Real-Time - Optimized Content

January 10,2019

Jirapongse Phuriphanvichai

Use Fiddler to Capture Refinitiv Real-Time - Optimized Content

Introduction

Refinitiv Real-Time - Optimized is a solution for applications to access Real-Time content via Cloud technologies. To use it, developers need to implement an application with REST API for authentication and WebSocket API for Real-Time content. With these technologies, accessing the Real-Time content from Cloud is quite easy. However, when observing problems, such as connection lost, incorrect content, or other unexpected behaviors, the application should have enough information in order to investigate the issue. Typically, the information required for an investigation is message tracing. There are three methods to trace the messages.

Implement a trace capability inside the application by logging all messages sent and received through REST API and WebSocket API
Enable tracing in the REST and WebSocket API libraries (However, the used libraries must support tracing)
Use a proxy to capture the incoming and outgoing messages

This article demonstrates the third method by using Fiddler as a proxy to capture all incoming and outgoing messages from REST and WebSocket APIs. It also shows how to modify the examples to connect to Refinitiv Real-Time Optimized through the Fiddler proxy. The modified examples are at https://github.com/Refinitiv-API-Samples/Article.RDP.WebSocket.MarketPriceEdpGwExamplesWithProxy.

Fiddler
- Configure Fiddler
Modify Examples to Use a Fiddler as a Proxy
- Python
- CSharp
- Java
View Fiddler Output
Save Fiddler Output
Troubleshooting
Conclusion

Fiddler

Fiddler is a web debugger proxy server application which can be used to capture HTTP(S) and Web Socket traffic and log it for review. It is a free tool which can be downloaded from https://www.telerik.com/fiddler. This article uses Fiddler version 5.0 for Windows.

Configure Fiddler

If you don't have Fiddler installed on the machine, please download Fiddler from https://www.telerik.com/fiddler, and then install it. This section demonstrates how to:

Configure Fiddler to decrypt the HTTP(S) traffic
Export and convert the Fiddler Root certificate file
Remove the Fiddler Root certificate

1. Configure Fiddler to decrypt the HTTPS traffic

Typically, Fiddler can capture the HTTP(S) traffic passed through it. However, the client may be unable to understand the data if the traffic has been encrypted. Fiddler can be configured to install the Fiddler Root certificate to intercept the HTTPS traffic and decrypt the data. Please follow these steps to configure Fiddler to capture and decrypt HTTP(S) traffic.

Run Fiddler application
Click Tools -> Options... from the menu
On the HTTPS tab, check both Capture HTTPS CONNECTs and Decrypt HTTPS traffic boxes

For the first setup, the dialog box may appear to inform about generating and trusting the Fiddler Root certificate. Click the Yes button

Click the Yes button again to install this certificate to the Windows certificate store

Click the Connections tab to verify the TCP port listened by Fiddler. The default port is 8888. Then click the OK button to close the Options window

Click File from the menu and verify that Capture Traffic is checked

After these steps, the Fiddler Root certificate has been added to the Windows certificate store. You can run the certmgr.msc command to verify the Fiddler Root certificate.

2. Export and convert the Fiddler Root certificate file

For some programming languages, the Fiddler Root certificate file is required when creating an encrypted connection through the Fiddler proxy. For example, in Python, the Fiddler Root certificate file is required when sending post requests and creating a web socket connection through the Fiddler proxy. For Java, the Fiddler Root certificate must be imported to the Java key store file. For .NET programming languages, this step can be ignored because the Fiddler Root certificate has already been imported to the Windows certificate store when configuring Fiddler to decrypt the HTTPS traffic.

Please follow these steps to export and convert the Fiddler Root certificate file to a format which can be used by Python and Java.

Run Fiddler application
Click Tools -> Options... from the menu
On the HTTPS tab, click the Actions button and select Export Root Certificate to Desktop to export the Fiddler Root certificate file to Desktop. The file FiddlerRoot.cer will be saved on the Desktop.

Click the OK button to close the Options window

Open the Command Prompt and run the following command to convert the Fiddler Root certificate file to the PEM standard. You can download the OpenSSL binaries from OpenSSL Binaries

openssl x509 -inform DER -in FiddlerRoot.cer -out FiddlerRoot.crt

The created FiddlerRoot.crt will be used in the following steps.

3. Remove the Fiddler Root certificate

After capturing the traffic, you may need to remove the installed Fiddler Root certificate from the Windows certificate store. Please follow these steps to remove the Fiddler Root certificate.

Run Fiddler application
Click Tools -> Options... from the menu
On the HTTPS tab, uncheck the Decrypt HTTPS traffic box

Click the Actions button and then select Remove Interception Certificates

Click the Yes button to confirm to delete the Fiddler Root certificate

Click the OK button to close the Options window

Then, you can run the certmgr.msc command to verify that the Fiddler Root certificate has been removed from the Trusted Root Certification Authorities

Modify Examples to Use a Fiddler as a Proxy

The CSharp, Java, and Python examples for connecting to the Refinitiv Real-Time Optimized are available in GitHub. They are console applications with command line arguments. This section demonstrates how to modify the examples to connect to the Refinitiv Real-Time Optimized through a proxy. In this section, the hostname/TCP port of a Fiddler proxy is 127.0.0.1:8888 and the certificate file is FiddlerRoot.crt created in the previous step.

The examples will be modified in the following points:

Setting a proxy when sending HTTP(S) post and get requests for authentication, token refresh, and service discovery
Setting a proxy when creating a web socket connection for getting market data

Python

There are two examples in the Python folder.

The market_price_edpgw_authentication.py example demonstrates authenticating via an HTTPS request to the RDP Gateway using a username and password. It then opens a WebSocket to the Refinitiv Real-Time Optimized service using the specified hostname, logs in with the retrieved token, and requests market content

The market_price_edpgw_service_discovery.py example demonstrates authenticating via an HTTPS request to the RDP Gateway using a username and password, and discovering endpoints of the Refinitiv Real-Time Optimized service via an HTTPS request to the Refinitiv Real-Time Optimized Service Discovery. The example then opens a WebSocket to an endpoint, logs in with the retrieved token, and requests market content

Please follow these steps to modify the examples to connect to RDP through a Fiddler proxy.

In the get_sts_token function of both examples, modify the requests.post function call to use a proxy and certificate file by using the proxies and verify arguments.

    	
                    r = requests.post(url,
                          headers={'Accept': 'application/json'},
                          data=data,
                          auth=(user, client_secret),
                          verify="FiddlerRoot.crt",
                          proxies={
                              'http':'http://127.0.0.1:8888',
                              'https':'http://127.0.0.1:8888'
                              })

The value of the verify argument is the location of the certificate file. In this case, the FiddlerRoot.crt file is in the working directory of the examples. The value of the proxies argument is proxy settings for the HTTP(S) connection.

2. In both examples, modify the code when running the web socket on a new thread by using the following kwargs arguments.

    	
                    wst = threading.Thread(target= ...,
                               kwargs={
                                   'http_proxy_host':'127.0.0.1',
                                   'http_proxy_port':'8888',
                                   'sslopt':{'ca_certs':'FiddlerRoot.crt'}
                                   })

The kwagrs parameter has the http_proxy_host and http_proxy_port arguments which contain the proxy host and port settings, and the ca_certs argument which contain the location of certificate file. In this case, the FiddlerRoot.crt file is in the working directory of the examples

3. In the market_price_edpgw_service_discovery.py example, modify the requests.get function call in the query_service_discovery function to use a proxy and certificate file by using the proxies and verify arguments.

    	
                    r = requests.get(url, 
                         headers={"Authorization": "Bearer " + sts_token}, 
                         params={"transport": "websocket"},
                         verify="FiddlerRoot.crt",
                         proxies={
                             'http':'http://127.0.0.1:8888',
                             'https':'http://127.0.0.1:8888'
                             })

I have modified the Python examples to connect through a proxy. The new optional arguments with default values are introduced for proxy settings and a certificate file.

Argument Name	Description	Default Value
--proxy_hostname	The hostname or IP address of the proxy	127.0.0.1
--proxy_port	The TCP port listened by the proxy	8888
--cert_file	The certificate file used by the proxy	FiddlerRoot.crt

The modified examples are available at MarketPriceEdpGwExamplesWithProxy.

CSharp

There are two examples in the CSharp folder.

MarketPriceEdpGwAuthenticationExample: Retrieves market-price content for TRI.N after authenticating with an authentication server and using tokens sent by that server to keep the connection alive

MarketPriceEdpGwServiceDiscoveryExample: Retrieves market-price content for TRI.N after authenticating with an authentication server and using tokens sent by that server to keep the connection alive and after using service discovery to find the location of the service providing the market-price content

Please follow these steps to modify the examples to connect to RDP through a Fiddler proxy.

1. In both examples, create a new WebProxy object with proxy settings.

    	
            WebProxy proxyObject = new WebProxy("http://127.0.0.1:8888/");

2. In the GetAuthenticationInfo method of both examples, assign the created WebProxy object (proxyObject) to the Proxy property of the HttpWebRequest

    	
            HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url); 
webRequest.Proxy = proxyObject;

3. In both examples, assign the created WebProxy object (proxyObject) to the Options.Proxy property of the ClientWebSocket object

    	
            WebSocket = new ClientWebSocket();

WebSocket.Options.Proxy = proxyObject;

4. In the MarketPriceEdpGwServiceDiscoveryExample example, assign the created WebProxy object (proxyObject) to the Proxy property of the HttpWebRequest in the DiscoverServices method

    	
            HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url); 
webRequest.Proxy = proxyObject;

C# examples don't require a certificate file because the WebSocket and web request libraries used by .NET programming languages can acquire the certificates installed in the Windows certificate store. Therefore, there is no need to specify it in the examples.

I have modified the C# examples to connect through a proxy. The new optional arguments with default values are introduced for proxy settings.

Argument Name	Description	Default Value
--proxy_hostname	The hostname or IP address of the proxy	127.0.0.1
--proxy_port	The TCP port listened by the proxy	8888

The modified examples are available at MarketPriceEdpGwExamplesWithProxy.

Java

There are two examples in the Java folder.

The MarketPriceEdpGwAuthentication example demonstrates authenticating via an HTTPS request to the RDP Gateway using a username and password. It then opens a WebSocket to the Refinitiv Real-Time Optimized service using the specified hostname, logs in with the retrieved token, and requests market content

The MarketPriceEdpGwServiceDiscovery example demonstrates authenticating via an HTTPS request to the RDP Gateway using a username and password, and discovering endpoints of the Refinitiv Real-Time Optimized service via an HTTPS request to the RDP Service Discovery. The example then opens a WebSocket to an endpoint, logs in with the retrieved token, and requests market content

Please follow these steps to modify the examples to connect to RDP through a Fiddler proxy.

1. In both examples, create a new HttpHost object with proxy settings and then build a custom RequestConfig with this proxy

    	
            HttpHost proxy = new HttpHost("127.0.0.1", 8888, "http"); 
RequestConfig config = RequestConfig.custom() 
    .setProxy(proxy) 
    .build();

2. In the GetAuthenticationInfo method of both examples, assign the created RequestConfig object (config) to the HttpPost object via the setConfig method

    	
            HttpPost httppost = new HttpPost(url); 
httppost.setConfig(config);

3. In both examples, configure proxy settings of the created WebSocketFactory object

    	
            WebSocketFactory factory = new WebSocketFactory(); 
factory.getProxySettings().setServer("http://127.0.0.1:8888");

4. In both examples, remove the code that adds a PERMESSAGE_DEFLATE extension when creating a websocket

    	
            _websocket = websocketFactory.createSocket(_url)
    .addProtocol("tr_json2")
    .addListener(new WebSocketAdapter()  
    ...
    });
    //.addExtension(WebSocketExtension.PERMESSAGE_DEFLATE);

If the PERMESSAGE_DEFLATE extension isn't removed, the WebSocket data shown in the Fiddler will be compressed.

5. In the queryServiceDiscovery method of the MarketPriceEdpGwServiceDiscovery example, assign the created RequestConfig object (config) to the HttpGet object via the setConfig method

    	
            HttpGet httpget = new HttpGet(new URIBuilder().setScheme("https").setHost(host).setPath(discoveryPath).setParameter("transport", "websocket").build());
httpget.setConfig(config);

Before running the examples, the Fiddler Root certificate must be imported to the Java Key Store. Typically, the Java Key Store file (cacerts) is in the JAVA_HOME\jre\lib\security\cacerts directory. The following command can be used to import the Fiddler Root certificate file.

    	
            C:\Program Files\Java\jdk1.8.0_131\jre\lib\security>keytool -import -alias fiddler -file FiddlerRoot.crt -keystore cacerts  -storepass changeit

The FiddlerRoot.crt file was created in the previous step (Export and convert the Fiddler Root certificate file).

The Fiddler Root certificate can be removed from the Java Key Store with the following command.

    	
            C:\Program Files\Java\jdk1.8.0_131\jre\lib\security>keytool -delete -alias fiddler -keystore cacerts -storepass changeit

I have modified the Java examples to connect through a proxy. The new optional arguments with default values are introduced for proxy settings.

Argument Name	Description	Default Value
--proxy_hostname	The hostname or IP address of the proxy	127.0.0.1
--proxy_port	The TCP port listened by the proxy	8888

The modified examples are available at MarketPriceEdpGwExamplesWithProxy.

View Fiddler Output

After configuring the Fiddler and modifying the examples to connect through the Fiddler, you can view the incoming and outgoing messages via the Fiddler application. When connecting to the Refinitiv Real-Time Optimized, you can view the following information.

1. Authentication and Refresh Token

The host and URL for authentication and refresh token are api.refinitivcom:443 and /auth/oauth2/v1/token, respectively. You can use Edit -> Find Sessions ... (Ctrl+F) to find this session.

Double click on the session to view the request and response messages. The decrypted request and response messages are displayed on the right panel.

For example:

Authentication

Refresh Token

2. Service Discovery

The host and URL for authentication and refresh token are api.refinitiv.com:443 and /streaming/pricing/v1/?transport=websocket, respectively. You can use Edit -> Find Sessions ... (Ctrl+F) to find this session.

Double click on the session to view the request and response messages. The decrypted request and response messages are displayed on the right panel.

For example:

3. Real-time Data via WebSocket The hostname for the streaming real-time data may depend on the configuration or service discovery. It looks like amer-3-t3.streaming-pricing-api.refinitiv.com:443. The URL is /WebSocket. You can use Edit -> Find Sessions ... (Ctrl+F) to find this session.

Double click on the session to view the request and response messages. The decrypted messages are displayed in the WebSocket tab on the right panel.

For example:

Save Fiddler Output

The output of the selected session can be saved by right-clicking on the selected session. Then select Save -> Selected Sessions -> in ArchiveZip...

The file type is Session Archive (.saz). The archived file can be re-opened via the Fiddler application by selecting the File -> Load Archive... menu.

Troubleshooting

1. Unable to connect to Fiddler proxy

When the examples are unable to connect to Fiddler proxy, the following exceptions should be displayed.

Python

    	
            Refinitiv Data Platform authentication exception failure: HTTPSConnectionPool(host='api.refinitiv.com', port=443): Max retries exceeded with url: /auth/oauth2/v1/token (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x000000000AF59080>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it',)))

CSharp

    	
            Authentication server request failed: No connection could be made because the target machine actively refused it No connection could be made because the target machine actively refused it -- No connection could be made because the target machine actively refused it

Java

    	
            Refinitiv Data Platform authentication failure:
org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:8888 [/127.0.0.1] failed: Connection refused: connect

To solve this issue, please verify that the Fiddler is configured and run properly and the examples use the correct IP address (--proxy_hostname) and TCP port (--proxy_port) configured in the Fiddler proxy.

2. The certificate is invalid or unavailable

If the certificate is invalid or unavailable, the following exceptions should be displayed.

Python

    	
            Refinitiv Data Platform authentication exception failure: HTTPSConnectionPool(host='api.refinitiv.com', port=443): Max retries exceeded with url: /auth/oauth2/v1/token (Caused by SSLErr
or(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)'),
))

To solve this issue, please make sure the --cert_file configuration is set with the converted Fiddler certificate file. The conversion process is available in the Export and convert the Fiddler Root certificate file section.

CSharp

    	
            Authentication server request failed: The SSL connection could not be established, see inner exception. The remote certificate is invalid according to the validation procedure. --The SSL connection could not be established, see inner exception.

To solve this issue, the Fiddler Root certificate file (FiddlerRoot.cer) must be imported to the Windows Trusted Root Certification Authorities.

Java

    	
            Refinitiv Data Platform authentication failure:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 
Refinitiv Data Platform authentication failure:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

To solve this issue, the Fiddler Root certificate (FiddlerRoot.crt) must be imported to the Java Key Store. For more information, please refer to the Modify Examples to Use a Fiddler as a Proxy -> Java section.

3. The captured Web Socket data is unreadable

The unreadable Web Socket data in Fiddler looks like:

To solve this issue, the compression extension must be removed when creating a Web Socket connection. For example, in the Java examples, the WebSocketExtension.PERMESSAGE_DEFLATE must be removed.

    	
            _websocket = websocketFactory.createSocket(_url)
   .addProtocol("tr_json2")
   .addListener(new WebSocketAdapter() 
   {
    ...                              
   });
   //.addExtension(WebSocketExtension.PERMESSAGE_DEFLATE);

Conclusion

This article demonstrates how to configure the Fiddler to capture the data when connecting to the Refinitiv Real-Time - Optimized. It explains how to modify the examples to connect to the Refinitiv Real-Time - Optimized via the Fiddler proxy.

The captured data can be decrypted and displayed on the Fiddler application. Moreover, it can be saved as a session archive, and then reopened via the Fiddler application.