Use mitmdump to capture Refinitiv Real-Time - Optimized content
Introduction
In the previous article: Use Fiddler to Capture Refinitiv Real-Time - Optimized Content, we introduced a Fiddler tool which is a web debugger proxy server application and can be used to capture HTTP(S) and Web Socket traffic. Although Fiddler is easy to use, it may not be a suitable tool in some scenarios. For example, Fiddler is a UI application so it requires a lot of resources to run. Moreover, after starting Fiddler, the program registers itself as the system proxy for Microsoft Windows Internet Services (WinInet), the HTTP layer used by Internet Explorer, Microsoft Office, and many other products. Therefore, in addition to traffic from the Refinitiv Real-Time Optimized, developers also see traffics from other products, such as web browsers, captured by Fiddler. Furthermore, this may cause unexpected behaviors in other products, such as VPN connections.
For this reason, in this article, we introduce mitmdump to capture HTTP(S) and WebSocket traffics. mitmdump is a command line tool so it requires less memory than Fiddler. Moreover, mitmdump doesn't interfere with the Windows system proxy so it will not affect other products.
mitmdump
mitmdump (Man In The Middle Dump) is a command-line tool in the mitmproxy package. It is a command line version of mitmproxy which is a free and open source interactive HTTPS proxy. It can be run on various operating systems including Windows, Linux, and macOS. Moreover, it can be run as a Docker image. To install mitmproxy, please refer to the mitmproxy official website. In this article, we only focus on the mitmdump tool on Windows.
Certificate Files
When the first time mitmdump is run, the certificate files will be created in the config directory (~/.mitmproxy or %USERPROFILE%/.mitmproxy by default). The files created by mitmdump in the .mitmproxy directory are as follows:
| mitmproxy-ca.pem | The certificate and the private key in PEM format. |
| mitmproxy-ca-cert.pem | The certificate in PEM format. Use this to distribute on most non-Windows platforms. |
| mitmproxy-ca-cert.p12 | The certificate in PKCS12 format. For use on Windows. |
| mitmproxy-ca-cert.cer | Same file as .pem, but with an extension expected by some Android devices. |
By default, the mitmdump uses this certificate to intercept encrypted connections. Moreover, these certificate files are required by the Refinitiv Real-Time Optimized applications when connecting through the mitmdump as a proxy. The examples for connecting to the Refinitiv Real-Time Optimized via a proxy server are available in GitHub.
- To run the Python example, the mitmproxy-ca-cert.pem file must be specified in the --cert_file argument
- To run the CSharp example, double click to the mitmproxy-ca-cert.p12 file to import the certificate to the Trusted Root Certification Authorities
- To run the Java example, use the keytool command to import the mitmproxy-ca-cert.pem file to the Java Key Store file (cacerts) which is in the JAVA_HOME\jre\lib\security\cacerts directory
C:\Program Files\Java\jdk1.8.0_131\jre\lib\security>keytool -import -alias mitmproxy -file mitmproxy-ca-cert.pem -keystore cacerts -storepass changeit
Then, the examples can be run with --proxy_hostname and --proxy_port arguments to connect to the mitmdump as a proxy.
Usages
This section lists usages of the mitmdump command from simpler to more complex.
- Run the mitmdump without any arguments
The mitmdump will be run with the default values. It will run as a regular proxy server listening at http://*:8080. It uses certificate files from the config directory (~/.mitmproxy or %USERPROFILE%/.mitmproxy). After running, it will display the full request URL with response status code for client connections.
C:\Program Files (x86)\mitmproxy\bin>mitmdump
Proxy server listening at http://*:8080
127.0.0.1:45223: clientconnect
127.0.0.1:45223: POST https://api.refinitiv.com/auth/oauth2/v1/token
<< 200 OK 2.56k
127.0.0.1:45225: clientconnect
127.0.0.1:45225: GET https://api.refinitiv.com/streaming/pricing/v1/?transport=websocket
<< 200 OK 708b
127.0.0.1:45227: clientconnect
127.0.0.1:45227: GET https://amer-3-t3.streaming-pricing-api.refinitiv.com/WebSocket
<< 101 Switching Protocols 0b
127.0.0.1:45227 -> WebSocket 1 message -> amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket
127.0.0.1:45227 <- WebSocket 1 message <- amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket
127.0.0.1:45227 -> WebSocket 1 message -> amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket
127.0.0.1:45227 <- WebSocket 1 message <- amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket
...
2. Run with the --list-port, or -p argument to change the listening port
By default, mitmdump is listening on TCP 8080. This TCP port can be changed via the --list-port or -p argument.
C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe -p 8888
Proxy server listening at http://*:8888
With the above command, mitmdump is listening on TCP 8888 so the application needs to use that port to connect to mitmdump.
3. Run with the --flow-detail argument to change the detail log level
The default log level of mitmdump is 1 which displays the full request URL with response status code. This can be changed via the --flow-detail argument. The maximum level is 3 (very verbose) which displays the full request URL with response status code, HTTP headers, full response content, the content of WebSocket and TCP messages.
C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe --flow-detail 3
Proxy server listening at http://*:8080
127.0.0.1:15129: clientconnect
127.0.0.1:15129: POST https://api.refinitiv.com/auth/oauth2/v1/token
Host: api.refinitiv.com
User-Agent: python-requests/2.18.4
Accept-Encoding: gzip, deflate
Accept: application/json
Connection: keep-alive
Content-Length: 127
Content-Type: application/x-www-form-urlencoded
...
grant_type: password
takeExclusiveSignOnControl: True
scope: trapi
<< 200 OK 2.36k
Date: Fri, 08 Feb 2019 07:43:58 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Content-Encoding: gzip
..
{
"access_token": "..."
...
127.0.0.1:15133 <- WebSocket 1 message <- amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket
[{"Type":"Ping"}]
127.0.0.1:15133 -> WebSocket 1 message -> amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket
{"Type": "Pong"}
...
4. Run with the --save-stream-file or -w argument and stream_websocket option to save stream flows to a file
By default, the log is displayed on the console. To save the content flow both HTTP(S) and WebSocket to a file, the following command can be used:
C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe --w content.log --flow-detail 3 --set stream_websocket=true
The content in the file is encoded. To view the content, the following command can be used:
C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe -r content.log --flow-detail 3
5. Pipe the mitmdump's output to another program to add timestamps to the output
The output from mitmdump doesn't have timestamps so to get timestamps, the output can be piped to another application to add timestamps to the output. The following is a Python 3 sample script which adds timestamps to the mitmdump's output and optionally creates a log file with timestamps.
import sys
import re
from datetime import datetime
fileObject = None
try:
if len(sys.argv) == 2:
fileObject=open(sys.argv[1],"w")
for line in sys.stdin:
if line.strip() != "":
if re.match("^(?:[0-9]{1,3}\.){3}[0-9]{1,3}:[0-9]*", line.strip()):
line = "\n"+datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3] + ": " + line
print(line)
if fileObject:
fileObject.write(line)
fileObject.flush()
except:
if fileObject:
fileObject.close()
The output from mitmdump must be piped to this script.
C:\Program Files (x86)\mitmproxy\bin>mitmdump --flow-detail 3 | python LogWithTime.py log.txt
The content is the log.txt file is:
Proxy server listening at http://*:8080
2021-02-08 17:15:51.477: 127.0.0.1:58603: clientconnect
2021-02-08 17:15:54.601: 127.0.0.1:58603: POST https://api.refinitiv.com/auth/oauth2/v1/token
Host: api.refinitiv.com
User-Agent: python-requests/2.18.4
Accept-Encoding: gzip, deflate
Accept: application/json
Connection: keep-alive
Content-Length: 127
...
grant_type: password
takeExclusiveSignOnControl: True
scope: trapi
<< 200 OK 2.57k
Date: Fri, 08 Feb 2021 10:15:54 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Content-Encoding: gzip
X-Amzn-Trace-Id: Root=1-5c5d56d9-a15a8ee094b8d03e69a1a590
X-Tr-Requestid: 309479a4-2892-4b59-ab54-6403e8c2856a
{
...
2021-02-08 17:16:18.618: 127.0.0.1:58607 <- WebSocket 1 message <- amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket
[{"Type":"Ping"}]
2021-02-08 17:16:18.622: 127.0.0.1:58607 -> WebSocket 1 message -> amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket
{"Type": "Pong"}
Summary
mitmdump is a useful tool which can be used as a web proxy to capture content from REST API and web socket. Therefore, it can be used to capture content from RDP. Mitmdump is a console application which is easy to use and supports multiple platforms including Windows, Linux, macOs, and Docker. This article shows how to use mitmdump to capture content from the Refinitiv Real-Time Optimized. The application needs to use mitmdurmp as a proxy when connecting to the Refinitiv Real-Time Optimized. The examples for connecting to the Refinitiv Real-Time Optimized via a proxy server are available in the GitHub. The output from mitmdump can be used to investigate issues, such an invalid credential, incorrect content, and disconnection.
