Introduction
Refinitiv Data Platform provides access to Refinitiv data set using RESTful API via Cloud technologies. The data set can be streaming or non-streaming of type request-response or bulk delivery or asynchronous alerts delivery. Often time, it is required that an application developer capture the raw HTTP request and response headers and payload for debugging purposes. Third party tools like Fiddler Proxy or mitmproxy can be helpful in capturing this information.
This article will discuss the use of Fiddler as a proxy, to capture the RDP data and also try to minimize the code changes required to do so. Please read this article for - Using a proxy to debug the streaming websocket data from Refinitiv Real-Time Optimized.
Setup and Configure Fiddler
Download and install Fiddler from https://www.telerik.com/fiddler. Fiddler is a proxy which can capture both regular web traffic (HTTP) and encrypted web traffic (HTTPS). A REST'ful application talking to a cloud service provider has to be configured to use a proxy, instead of a direct connection. Since the HTTPS connection is encrypted, Fiddler has to be configured to install the Fiddler Root certificate to intercept the traffic and decrypt the data. This is a potentially dangerous technique and should only be done on a machine used for development purposes.
To Configure Fiddler to decrypt the HTTPS traffic:
- Run Fiddler application
- Click Tools -> Options... from the menu
- On the HTTPS tab, check both Capture HTTPS CONNECTs and Decrypt HTTPS traffic boxes
- A dialog box will appear which will inform about generating and trusting the Fiddler Root certificate. Click the Yes button
- Once the certificate has been generated, click YES to add the certificate to the Windows Certificate store.
At this point, most application which use and trust Windows certificate store, like Microsoft Edge or Chrome browser etc, will start using Fiddler for HTTP and HTTPS traffic transparently. Some applications like Mozilla Firefox, however have their own certificate stores and will issue a security warning if their traffic is routed through Fiddler. For these applications, the Fiddler certificate has to be manually added into the application's trusted store.
Let us explore the use of proxy with common programming languages.
Export and store certificate
Once the root certificate has been generated in the previous step, and the proxy server is running, we have to inform our applications to start routing the web traffic through this proxy. The built in libraries in most programming languages, easily allow specifying the proxy information through the use of environment variables. Some libraries, like Microsoft .NET framework automatically read and use the proxy information from operating system settings. Its usually the third party libraries which are troublesome to configure, and require proxy configuration to be explicitly passed in the code. These libraries and languages are identified below.
First step in this process is to export the proxy root certificate so that we can make it available to the applications.
Export and convert the Fiddler Root certificate file
To export and convert the Fiddler Root certificate file, follow these steps:
- Run Fiddler application
- Click Tools -> Options
- On the HTTPS tab, click the Actions button and select Export Root Certificate to Desktop. The file FiddlerRoot.cer will be saved on the Desktop.
Some applications like Python require the certificate in PEM format. Our exported CER file can be converted using OpenSSL using following command:
openssl x509 -inform DER -in FiddlerRoot.cer -out FiddlerRoot.pem
Now, all is left is to make this certificate available to our applications and get them to trust it.
Language specific configuration
Although many applications and hence language implementations will automatically pick up the proxy configuration from OS; some languages have to be explicitly asked to use a proxy server. This is done either though the use of environment variables or via changes to the application source code. The following examples assume that Fiddler is running on the local machine and listening for HTTP and HTTPS traffic on port 36036. Also, the FiddlerRoot certificate is stored in some accessible path on local machine. Only the most common library is considered in the language specific examples below.
Jump to Languages/tools section:
Python
Python requests is the most commonly used library for REST API programming. requests reads and respects the environment variables which specify the proxy information. This would suffice for capturing plain HTTP traffic. However, these requests will fail with a security exception for a HTTPS connection. This is because the response from a proxy server is signed with an untrusted (Fiddler) certificate. To circumvent this issue, Fiddler Certificate Authority has to be passed to both requests and websocket libraries.
Setting the following environment variables before invoking the Python application does the trick:
set https_proxy=http://127.0.0.1:36036
set http_proxy=http://127.0.0.1:36036
set REQUESTS_CA_BUNDLE=<path to FiddlerRoot.pem>
set WEBSOCKET_CLIENT_CA_BUNDLE=<path to FiddlerRoot.pem>
Typical Python code using requests for REST API will be used like:
import requests
.
.
dResp = requests.get(RESOURCE_ENDPOINT, headers = {"Authorization": "Bearer " + accessToken}, params = requestData);
And now all HTTP(S) request/responses from a Python application will be captured by Fiddler:
NodeJS
In NodeJS, request is the most commonly used HTTP/REST library. A NodeJS example, using request API will be along the following lines:
const request = require("request");
const httpOptions = {
url: `https://api.refinitiv.com/data/environmental-social-governance/v1/views/basic?universe=${RIC}`,
headers: {
Authorization: "Bearer " + accessToken
}
};
request.get(httpOptions, function(error, response, body) {
if(!error && response.statusCode == 200)
resolve(JSON.parse(body));
else
reject(error);
By default, request honors the proxy information passed in the environment variables. This will work for plain HTTP traffic, but not for HTTPS capture. For HTTPS traffic, either the application has to be instructed to trust the Fiddler Root Certificate, or an easier options is to ask NodeJS to ignore the SSL certificate errors (remember this is being done on a development machine!).
Set the following environment variables before starting the Node application:
set https_proxy=http://127.0.0.1:36036
set http_proxy=http://127.0.0.1:36036
set NODE_TLS_REJECT_UNAUTHORIZED=0
and now the NodeJS application will use these variables, and all the web traffic will be routed through and captured by our proxy server, as shown:
Java
For Java, the Apache HttpComponents are the most commonly used library for using the REST API. A sample code to invoke a RDP endpoint and get data from it would look something like:
HttpClient httpclient = myHttpClientBuilder.build();
// create a Post message
HttpPost request = new HttpPost(tokenEndpoint);
// set the data payload
request.addHeader("content-type", "application/x-www-form-urlencoded");
request.addHeader("Authorization", authBasic);
request.setEntity(params);
// send the request. Request is channeled through the proxy server
HttpResponse response = httpClient.execute(request);
// read the response from server
StatusLine status = response.getStatusLine();
Here a POST request is being sent to a server, which includes the Authorization token in the header, and the server response is then checked for proper status code. The complete code for snippet shown above is available in the reference section. In its default usage as shown here, the HttpClient has to be explicitly set to use a proxy server, by passing the proxy information in the Java code. A better approach is to instantiate the HttpClient, with properties set in the environment variables. Doing so, we can control and use/disuse the proxy server without making any code changes.
To get HttpClient to respect the proxy information passed in the Java properties, it has to be instantiated in the following manner:
// original:
// HttpClient httpclient = myHttpClientBuilder.build();
// new: include system properties before build call
HttpClient httpclient = myHttpClientBuilder.useSystemProperties().build();
This change will ensure that command line proxy information is respected by the Apache library. At this point this application would be functional, if the REST API were using plain HTTP and not the encrypted HTTPS protocol. Few additional changes are required to make it work with SSL transport. The Java runtime has to be told to trust the Fiddler's root certificate. Not doing so, will throw an "untrusted host exception".
Use the following steps to create a Java keystore file with the previously exported Fiddler certificate:
keytool.exe -import -file <path to FiddlerRoot.cer> -keystore FiddlerKeystore -alias Fiddler
And now, invoke the Java application with these additional properties on the command line:
-DproxySet=true -DproxyHost=127.0.0.1 -DproxyPort=36036 -Djavax.net.ssl.trustStore=FiddlerKeystore -Djavax.net.ssl.trustStorePassword=<Keystore Password>
Similarly, other third party libraries for Java will have their own specific way to configure the proxy server.
Snapshot showing Java application and requests being captured in Fiddler:
C#/.NET
The .NET example for getting data from RDP uses an additional library for REST API calls. The Microsoft provided Microsoft.AspNet.WebApi.Client offers convenient methods to invoke HTTP GET and POST requests. This library can be installed using NuGet.
Here is a sample Code which gets an authentication token using POST method:
using System.Net.Http;
using Newtonsoft.Json.Linq;
.
.
private HttpClient client;
public async Task<bool> requestToken() {
// request for getting an access token
var formContent = new FormUrlEncodedContent(new[] {
new KeyValuePair<string, string>("grant_type", "password"),
new KeyValuePair<string, string>("username", username),
.
.
// add client id as basic authentication
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", Convert.ToBase64String(System.Text.ASCIIEncoding.ASCII.GetBytes(string.Format("{0}:{1}", clientID, clientSecret))));
// send the request
HttpResponseMessage response = client.PostAsync(tokenEndpoint, formContent).Result;
if (response.IsSuccessStatusCode) {
.
This System.Net.Http library which is part of WebApi.Client package used in the sample code here, automatically picks up the proxy configuration from the operating system. Since the Fiddler application registers itself as a system proxy when it starts up, no additional action needs to be taken to capture the packets.
Running our C#/.NET application, while Fiddler is running results in:
Postman:
The Postman application is commonly used to test out the REST API's. Just like .NET application, Postman automatically uses the proxy information from the operating system, so no additional configuration in needed.
Running the RDP API samples, we can see the captured request and response messages:
References:
Refinitiv Data API - https://developers.refinitiv.com/refinitiv-data-platform/refinitiv-data-platform-apis
Fiddler - https://www.telerik.com/fiddler
