Use mitmdump to Capture Refinitiv Real-Time - Optimized Content

Article

January 30, 2019

Jirapongse Phuriphanvichai

Developer Advocate

Use mitmdump to capture Refinitiv Real-Time - Optimized content

Introduction

In the previous article: Use Fiddler to Capture Refinitiv Real-Time - Optimized Content, we introduced a Fiddler tool which is a web debugger proxy server application and can be used to capture HTTP(S) and Web Socket traffic. Although Fiddler is easy to use, it may not be a suitable tool in some scenarios. For example, Fiddler is a UI application so it requires a lot of resources to run. Moreover, after starting Fiddler, the program registers itself as the system proxy for Microsoft Windows Internet Services (WinInet), the HTTP layer used by Internet Explorer, Microsoft Office, and many other products. Therefore, in addition to traffic from the Refinitiv Real-Time Optimized, developers also see traffics from other products, such as web browsers, captured by Fiddler. Furthermore, this may cause unexpected behaviors in other products, such as VPN connections.

For this reason, in this article, we introduce mitmdump to capture HTTP(S) and WebSocket traffics. mitmdump is a command line tool so it requires less memory than Fiddler. Moreover, mitmdump doesn't interfere with the Windows system proxy so it will not affect other products.

mitmdump

mitmdump (Man In The Middle Dump) is a command-line tool in the mitmproxy package. It is a command line version of mitmproxy which is a free and open source interactive HTTPS proxy. It can be run on various operating systems including Windows, Linux, and macOS. Moreover, it can be run as a Docker image. To install mitmproxy, please refer to the mitmproxy official website. In this article, we only focus on the mitmdump tool on Windows.

Certificate Files

When the first time mitmdump is run, the certificate files will be created in the config directory (~/.mitmproxy or %USERPROFILE%/.mitmproxy by default). The files created by mitmdump in the .mitmproxy directory are as follows:

mitmproxy-ca.pem	The certificate and the private key in PEM format.
mitmproxy-ca-cert.pem	The certificate in PEM format. Use this to distribute on most non-Windows platforms.
mitmproxy-ca-cert.p12	The certificate in PKCS12 format. For use on Windows.
mitmproxy-ca-cert.cer	Same file as .pem, but with an extension expected by some Android devices.

By default, the mitmdump uses this certificate to intercept encrypted connections. Moreover, these certificate files are required by the Refinitiv Real-Time Optimized applications when connecting through the mitmdump as a proxy. The examples for connecting to the Refinitiv Real-Time Optimized via a proxy server are available in GitHub.

To run the Python example, the mitmproxy-ca-cert.pem file must be specified in the --cert_file argument
To run the CSharp example, double click to the mitmproxy-ca-cert.p12 file to import the certificate to the Trusted Root Certification Authorities
To run the Java example, use the keytool command to import the mitmproxy-ca-cert.pem file to the Java Key Store file (cacerts) which is in the JAVA_HOME\jre\lib\security\cacerts directory

Then, the examples can be run with --proxy_hostname and --proxy_port arguments to connect to the mitmdump as a proxy.

Usages

This section lists usages of the mitmdump command from simpler to more complex.

Run the mitmdump without any arguments

The mitmdump will be run with the default values. It will run as a regular proxy server listening at http://*:8080. It uses certificate files from the config directory (~/.mitmproxy or %USERPROFILE%/.mitmproxy). After running, it will display the full request URL with response status code for client connections.

2. Run with the --list-port, or -p argument to change the listening port

By default, mitmdump is listening on TCP 8080. This TCP port can be changed via the --list-port or -p argument.

With the above command, mitmdump is listening on TCP 8888 so the application needs to use that port to connect to mitmdump.

3. Run with the --flow-detail argument to change the detail log level

The default log level of mitmdump is 1 which displays the full request URL with response status code. This can be changed via the --flow-detail argument. The maximum level is 3 (very verbose) which displays the full request URL with response status code, HTTP headers, full response content, the content of WebSocket and TCP messages.

4. Run with the --save-stream-file or -w argument and stream_websocket option to save stream flows to a file

By default, the log is displayed on the console. To save the content flow both HTTP(S) and WebSocket to a file, the following command can be used:

The content in the file is encoded. To view the content, the following command can be used:

5. Pipe the mitmdump's output to another program to add timestamps to the output

The output from mitmdump doesn't have timestamps so to get timestamps, the output can be piped to another application to add timestamps to the output. The following is a Python 3 sample script which adds timestamps to the mitmdump's output and optionally creates a log file with timestamps.

The output from mitmdump must be piped to this script.

The content is the log.txt file is:

Summary

mitmdump is a useful tool which can be used as a web proxy to capture content from REST API and web socket. Therefore, it can be used to capture content from RDP. Mitmdump is a console application which is easy to use and supports multiple platforms including Windows, Linux, macOs, and Docker. This article shows how to use mitmdump to capture content from the Refinitiv Real-Time Optimized. The application needs to use mitmdurmp as a proxy when connecting to the Refinitiv Real-Time Optimized. The examples for connecting to the Refinitiv Real-Time Optimized via a proxy server are available in the GitHub. The output from mitmdump can be used to investigate issues, such an invalid credential, incorrect content, and disconnection.