Author:
Refinitiv Real-Time Optimized (RTO) is the Refinitiv conflated real-time service, hosted in the public cloud. It provides fast and simple access to our unparalleled content from hundreds of exchanges and OTC markets around the world. We can use WebSocket APIs or Real-Time SDKs to connect to Real-Time Optimized to consume real-time content. Refinitiv Real-Time Optimized is maintained by Refinitiv so we don’t have control over it. It means we are unable to shut down the services to test how applications handle unexpected behaviors originated by Real-Time Optimized or Refinitiv Data Platform services.
This article demonstrates how to set up a simulated Real-Time Optimized in 3 steps to test Refinitiv Real-Time SDK’s applications. It also shows how to configure and run EMA examples to connect to this simulated Real-Time Optimized environment.
Note: This simulated environment is for development and testing purposes only.
Prerequisites
To follow the steps in this article the following applications are required.
1. OpenSSL toolkit
2. Node.js
3. An RSSL interactive provider
Simulated Real-Time Optimized Environment
Refinitiv Real-Time Optimized is comprised of three secured services in the public cloud. The first two services are HTTPS REST services on the Refinitiv Data Platform for authentication and service discovery and the third service is a secured streaming service that provides financial streaming content. This article shows how to use a node application and an RSSL interactive provider application to simulate HTTPS REST services and a streaming service, respectively.
The following picture shows how the Refinitiv Real-Time SDK consumer applications connect to Refinitiv Real-Time Optimized and the simulated Real-Time Optimized environment.
There are three steps to set up a Simulated Real-Time Optimized Environment. To simplify the setup process, the Node.js application and RSSL interactive provider are running on the same machine.
1. Generate a certificate file and a private key file
Refinitiv Real-Time Optimized uses encrypted connections so we need to have a certificate file and a private key file for the simulated Real-Time Optimized environment. You may need to contact your IT team to generate those files for you. Otherwise, you can use the OpenSSL command to generate those files. This is for development/testing purposes only.
Run the following command to generate a server certificate file and a server private key file for the www.test.com domain.
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -subj '/CN=www.test.com'
The key.pem file is the private key file and the cert.pem file is the certificate file.
Then, run the following command to remove the PEM pass phrase from the key.pem file.
cp key.pem key.pem.org
openssl rsa -in key.pem.org -out key.pem
The key.pem and cert.pem files will be used by the Node.js and RSSL interactive provider application and the cert.pem file will be used by Refinitiv Real-time SDK consumer applications to connect to this simulated Real-Time Optimized environment.
The key.pem and cert.pem files used by this article are also available on GitHub. The description of the cert.pem file on GitHub is:
Common Name : www.test.com
Valid From : Sep 30,2022
Valid To : Sep 27,2032
Serial Number : 18138038264291739402
2. Run a Node.js application
We used a Node.js application with the Express framework to simulate REST services on the Refinitiv Data Platform. The example code is available on GitHub.
First, the application uses the private key (key.pem) and certificate (cert.pem) files created in the first step to start the HTTPS server on TCP port 433.
const https = require('https');
const fs = require('fs');
const bodyParser = require('body-parser');
const listen_port = 443
const app = express();
app.use(bodyParser.urlencoded({ extended: true }));
https.createServer(
{
key: fs.readFileSync("key.pem"),
cert: fs.readFileSync("cert.pem"),
},
app
)
.listen(listen_port, () => {
console.log(timestamp()+"server is running at port "+ listen_port);
});
You can change to other TCP ports.
Next, the application simulates the following Refinitiv Data Platform services.
· RDP Authentication Service V1
The endpoint of this service is auth/oauth2/v1/token. It is used to get and renew tokens. The application receives HTTP Post requests from this endpoint and then sends successful or failed responses.
const success_token = `{
"access_token": "eyJ0eXAiOiJhdCtqd3QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImJlcGpHV0dkOW44WU9VQ1Nw",
"refresh_token": "aa957c85-f1f2-453d-8211-adbbb30b8428",
"expires_in": "600",
"scope": "trapi.data.pricing.read trapi.streaming.pricing.read",
"token_type": "Bearer"
}`
const bad_request_token = '{"error": "invalid_grant"}'
app.post('/auth/oauth2/v1/token', (req, res) => {
console.log(timestamp()+req.url)
console.log("grant_type: "+req.body.grant_type)
console.log(req.body)
if (req.body.grant_type == 'password') {
res.send(success_token)
} else if (req.body.grant_type == 'refresh_token') {
res.send(success_token)
//console.log("Send 400 Bad Request")
//res.status(400).send(bad_request_token)
} else {
console.log("Unknown grant_type")
res.status(400).send(bad_request_token)
}
})
We can simulate failed requests by sending the HTTP 400 Bad Request response status code with the error message in the HTTP body.
· RDP Authentication Service V2
The endpoint of this service is auth/oauth2/v2/token. It is used to get an access token. The application receives HTTP Post requests from this endpoint and then sends successful or failed responses.
const success_token_v2 = `{
"access_token": "eyJ0eXAiOiJhdCtqd3QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImJlcGpHV0dkOW44WU9VQ1Nw",
"expires_in": "7199",
"scope": "trapi.data.pricing.read trapi.streaming.pricing.read",
"token_type": "Bearer"
}`
app.post('/auth/oauth2/v2/token', (req, res) => {
console.log(timestamp() + req.url)
console.log("grant_type: " + req.body.grant_type)
console.log(req.body)
if (req.body.grant_type == 'client_credentials') {
res.send(success_token_v2)
} else {
console.log("Unknown grant_type")
res.status(400).send(bad_request_token)
}
})
We can simulate failed requests by sending the HTTP 400 Bad Request response status code with the error message in the HTTP body.
· Streaming Service Endpoint Information
The endpoint of this service is /streaming/pricing/v1/. It is used to get available streaming servers. The Refinitiv Real-Time SDK consumer applications will connect to one of these streaming servers to retrieve conflated real-time content. The application receives HTTP GET requests from this endpoint and then sends successful responses that contain all available streaming servers.
const endpoints_data = fs.readFileSync('endpoints.json', 'utf8');
app.get('/streaming/pricing/v1/', (req, res) => {
console.log(timestamp()+req.url)
res.send(endpoints_data)
})
The application loads the response from the local file (endpoints.json) and then sends it in the HTTP response. The content of the endpoints.json looks like this:
{
"services": [
{
"port": 14002,
"location": [
"ap-northeast-1a"
],
"transport": "tcp",
"provider": "aws",
"endpoint": "www.test.com",
"dataFormat": [
"rwf"
]
},
{
"port": 14002,
"location": [
"ap-northeast-1a",
"ap-northeast-1b"
],
"transport": "tcp",
"provider": "aws",
"endpoint": "www.test.com",
"dataFormat": [
"rwf"
]
},
…
The response informs the Refinitiv Real-Time SDK consumer applications to connect to www.test.com on TCP port 14002 to retrieve conflated real-time content. You can modify the hostname in the “endpoint” property.
The source code is available on GitHub. Download the source code and run “npm install” to install dependencies.
You can start the Node.js application by running the following command.
node app.js
The output looks like this:
[root@www rdpservices]# node app.js
RDP Services Simulator
[6:6:29] server is running at port 443
3. Run an RSSL interactive provider application
We used an ETA Provider example (Cpp-C\Eta\Applications\Examples\Provider) in the Refinitiv Real-Time SDK C/C++ package to simulate a Real-Time Optimized feed. The example generates dummy real-time data and sends it to the consumer application.
UpdateMsg
streamId="5"
domain="MarketPrice Domain"
updateTypeNum="0"
name="IBM.N"
serviceId="1"
serviceName="ELEKTRON_DD"
Payload dataType="FieldList"
FieldList
FieldEntry fid="6" name="TRDPRC_1" dataType="Real" value="1.09"
FieldEntry fid="22" name="BID" dataType="Real" value="1.08"
FieldEntry fid="25" name="ASK" dataType="Real" value="203.12"
FieldEntry fid="32" name="ACVOL_1" dataType="Real" value="100000.00"
FieldEntry fid="11" name="NETCHNG_1" dataType="Real" value="203.15"
FieldEntry fid="267" name="ASK_TIME" dataType="Time" value="08:55:54:000:000:000"
FieldListEnd
PayloadEnd
UpdateMsgEnd
You can modify the code to test unexpected behavior, such as sending a login closed status message.
Run the example with the following arguments. We specify the key and certificate files created in the first step in the arguments.
./Provider -c encrypted -s ELEKTRON_DD -keyfile key.pem -cert cert.pem -p 14002 -runtime 99999999
The output looks like this:
[root@www providers]# ./Provider -c encrypted -s ELEKTRON_DD -keyfile key.pem -cert cert.pem -p 14002 -runtime 99999999
portNo: 14002
serviceName: ELEKTRON_DD
serviceId: 1
Server IPC descriptor = 3 bound on port 14002
Instead of using an ETA interactive provider example, you can run an encrypted ADS or testserver tool, as a simulated Real-time Optimized feed. Testserver is a test tool in the Refinitiv Real-Time Distribution System package. You can run the testserver tool with the following arguments. We specify the key and certificate files created in the first step in the arguments.
./testserver -S ELEKTRON_DD -N 14002 -K -U 1 -serverCert cert.pem -serverPrivateKey key.pem -ct rssl -rct 1
To set up an encrypted ADS, please refer to the Configuring Encrypted Connections on ADS article.
In summary, the settings for connecting to this simulated Refinitiv Real-Time Optimized are:
| Service | URL | TCP Port |
|---|---|---|
| RDP Authentication Service V1 | www.test.com/auth/oauth2/v1/token | 443 |
| RDP Authentication Service V2 | www.test.com/auth/oauth2/v2/token | 443 |
| Streaming Service Endpoint Information (Service Discovery) | www.test.com/streaming/pricing/v1/ | 443 |
| Real-Time Streaming | www.test.com | 14002 |
Run EMA Examples
In this section, I will demonstrate how to configure and run EMA C++ and Java examples on a Windows machine to connect to the simulated Real-Time Optimized environment running on the same machine (localhost).
Run the Enterprise Message API C++ Example (Cons450)
450_MP_QueryServiceDiscovery (Cons450) is an OMM Consumer application example that demonstrates basic usage of the EMA library in accessing and parsing OMM MarketPrice data from Refinitiv Real-Time - Optimized. It illustrates how to query endpoints from RDP service discovery using the ServiceEndpointDiscovery class and use the location from the command line to select an endpoint. It uses the EMA's programmatic configuration to enable session management with the retrieved endpoint for establishing a connection with the Refinitiv Real-Time service and consuming data.
Please follow these steps to run the Cons450 example to connect to the simulated Real-Time Optimized environment.
1. Setup the Windows host file
The key and certificate files used by the node.js and RSSL interactive provider are created by using www.test.com as a common name and those applications are running on the same Windows machine as the Cons113 example. Therefore, to connect to those applications, we need to add the following entry into the Windows host file (C:\Windows\System32\drivers\etc\hosts) to resolve www.test.com to 127.0.0.1.
127.0.0.1 www.test.com
Note: The administrator privilege is required to add a new entry into the Windows host file.
2. Import a certificate file
To run the example, the certificate file (cert.pem) must be imported to the Windows Trusted Root Certification Authorities.
o Run certmgr to open the Manage Computer Certificates window
o Right click Trusted Root Certification Authorities to import the certificate file (cert.pem)
o Select the cert.pem file.
You can check that the certificate file is added in the Trusted Root Certification Authorities.
Note: The administrator privilege is required to add a new entry into the Windows host file.
3. Run the example
Run the Cons450 example with the following parameters.
o Use RDP Authentication Service V1
-username <machine id> -password <password> -clientId <client id> -itemName IBM.N -tokenURLV1 https://www.test.com/auth/oauth2/v1/token -serviceDiscoveryURL https://www.test.com/streaming/pricing/v1/
o Use RDP Authentication Service V2
-clientId <client id> -clientSecret <client secret> -itemName IBM.N -tokenURLV2 https://www.test.com/auth/oauth2/v2/token -serviceDiscoveryURL https://www.test.com/streaming/pricing/v1/
Note: You can use dummy credentials to connect to this simulated environment.
The output looks like this:
Services :
Service :
Provider : aws
Transport : tcp
Endpoint : www.test.com
Port : 14002
Data Format : rwf
Location : ap-northeast-1a
Service :
Provider : aws
Transport : tcp
Endpoint : www.test.com
Port : 14002
Data Format : rwf
Location : ap-northeast-1a ap-northeast-1b
Service :
Provider : aws
Transport : tcp
Endpoint : www.test.com
Port : 14002
Data Format : rwf
Location : ap-northeast-1b
…
…
RefreshMsg
streamId="5"
domain="MarketPrice Domain"
Solicited
RefreshComplete
ClearCache
state="Open / Ok / None / 'Item Refresh Completed'"
itemGroup="00 00"
qos="RealTime/TickByTick"
name="IBM.N"
nameType="1"
serviceId="1"
serviceName="ELEKTRON_DD"
Payload dataType="FieldList"
FieldList
FieldEntry fid="2" name="RDNDISPLAY" dataType="UInt" value="100"
FieldEntry fid="4" name="RDN_EXCHID" dataType="Enum" value="155"
FieldEntry fid="38" name="DIVPAYDATE" dataType="Date" value="22 OCT 2010"
FieldEntry fid="6" name="TRDPRC_1" dataType="Real" value="1.00"
FieldEntry fid="22" name="BID" dataType="Real" value="0.99"
FieldEntry fid="25" name="ASK" dataType="Real" value="203.03"
FieldEntry fid="32" name="ACVOL_1" dataType="Real" value="100000.00"
FieldEntry fid="11" name="NETCHNG_1" dataType="Real" value="203.15"
FieldEntry fid="267" name="ASK_TIME" dataType="Time" value="15:55:38:000:000:000"
FieldEntry fid="14305" name="EVENT_T_NS" dataType="Time" value="12:14:07:966:111:999"
FieldListEnd
PayloadEnd
RefreshMsgEnd
UpdateMsg
streamId="5"
domain="MarketPrice Domain"
updateTypeNum="0"
name="IBM.N"
serviceId="1"
serviceName="ELEKTRON_DD"
Payload dataType="FieldList"
FieldList
FieldEntry fid="6" name="TRDPRC_1" dataType="Real" value="1.01"
FieldEntry fid="22" name="BID" dataType="Real" value="1.00"
FieldEntry fid="25" name="ASK" dataType="Real" value="203.04"
FieldEntry fid="32" name="ACVOL_1" dataType="Real" value="100000.00"
FieldEntry fid="11" name="NETCHNG_1" dataType="Real" value="203.15"
FieldEntry fid="267" name="ASK_TIME" dataType="Time" value="15:55:39:000:000:000"
FieldEntry fid="14305" name="EVENT_T_NS" dataType="Time" value="12:14:07:966:111:999"
FieldListEnd
PayloadEnd
UpdateMsgEnd
…
…
Run the Enterprise Message API Java Example (ex450_MP_QueryServiceDiscovery)
ex450_MP_QueryServiceDiscovery is an OMM Consumer application example that demonstrates basic usage of the EMA library in accessing and parsing OMM MarketPrice data from Refinitiv Real-Time - Optimized. It illustrates how to query endpoints from Refinitiv Data Platform service discovery using the ServiceEndpointDiscovery class and use the location from the command line to select an endpoint. It uses the EMA's programmatic configuration to enable session management with the retrieved endpoint for establishing a connection with the Refinitiv Real-Time - Optimized service and consuming data.
Please follow these steps to run the ex450_MP_QueryServiceDiscovery example to connect to the simulated Real-Time Optimized environment.
1. Setup the Windows host file
The key and certificate files used by the node.js and RSSL interactive provider are created by using www.test.com as a common name and those applications are running on the same Windows machine as the ex450_MP_QueryServiceDiscovery example. Therefore, to connect to those applications, we need to add the following entry into the Windows host file (C:\Windows\System32\drivers\etc\hosts) to resolve www.test.com to 127.0.0.1.
127.0.0.1 www.test.com
Note: The administrator privilege is required to add a new entry into the Windows host file.
2. Import a certificate file
To run the example, the certificate file (cert.pem) must be imported to the Java trust store file. For example, the Java trust store file for JDK 11 is at C:\Program Files\Java\jdk-11.0.11\lib\security\cacerts.
o Copy the certificate file (cert.pem) to C:\Program Files\Java\jdk-11.0.11\lib\security\
o Run a command prompt as administrator and change the directory to C:\Program Files\Java\jdk-11.0.11\lib\security\
o Run the following command to import the certificate file (cert.pem) to the Java trust store file. “changeit” is the default password of the Java trust store file.
keytool -import -alias www.test.com -keystore cacerts -file cert.pem -storepass changeit
You can run the following command to see the imported certificate file.
keytool -list -alias www.test.com -keystore cacerts -storepass changeit
You can delete it by using the following command.
keytool -delete -alias www.test.com -keystore cacerts -storepass changeit
3. Run the example
Run the ex450_MP_QueryServiceDiscovery.Consumer example with the following parameters.
o Use RDP Authentication Service V1
-username <machind id> -password <password> -clientId <client id> -itemName IBM.N -tokenURLV1 https://www.test.com/auth/oauth2/v1/token -serviceDiscoveryURL https://www.test.com/streaming/pricing/v1/
o Use RDP Authentication Service V2
-clientId <client id> -clientSecret <client secret> -itemName IBM.N -tokenURLV2 https://www.test.com/auth/oauth2/v2/token -serviceDiscoveryURL https://www.test.com/streaming/pricing/v1/
Note: You can use dummy credentials to connect to this simulated environment.
The output looks like this:
Services :
Service :
Provider : aws
Transport : tcp
Endpoint : www.test.com
Port : 14002
Data Format : rwf
Location : ap-northeast-1a
Service :
Provider : aws
Transport : tcp
Endpoint : www.test.com
Port : 14002
Data Format : rwf
Location : ap-northeast-1a ap-northeast-1b
…
Oct 21, 2022 12:52:43 PM com.refinitiv.ema.access.ChannelCallbackClient reactorChannelEventCallback
INFO: loggerMsg
ClientName: ChannelCallbackClient
Severity: Info
Text: Received ChannelUp event on channel Channel_1
Instance Name Consumer_1_1
Component Version eta3.6.6.L1.win.rrg 64-bit Static
loggerMsgEnd
RefreshMsg
streamId="5"
domain="MarketPrice Domain"
solicited
RefreshComplete
state="Open / Ok / None / 'Item Refresh Completed'"
itemGroup="00 00"
name="PTT.BK"
nameType="1"
serviceId="1"
serviceName="ELEKTRON_DD"
Payload dataType="FieldList"
FieldList
FieldEntry fid="2" name="RDNDISPLAY" dataType="UInt" value="100"
FieldEntry fid="4" name="RDN_EXCHID" dataType="Enum" value="155"
FieldEntry fid="38" name="DIVPAYDATE" dataType="Date" value="22 OCT 2010"
FieldEntry fid="6" name="TRDPRC_1" dataType="Real" value="1.00"
FieldEntry fid="22" name="BID" dataType="Real" value="0.99"
FieldEntry fid="25" name="ASK" dataType="Real" value="203.03"
FieldEntry fid="32" name="ACVOL_1" dataType="Real" value="100000.00"
FieldEntry fid="11" name="NETCHNG_1" dataType="Real" value="203.15"
FieldEntry fid="267" name="ASK_TIME" dataType="Time" value="12:52:43:000:000:000"
FieldEntry fid="14305" name="EVENT_T_NS" dataType="Time" value="12:14:07:966:111:999"
FieldListEnd
PayloadEnd
RefreshMsgEnd
UpdateMsg
streamId="5"
domain="MarketPrice Domain"
updateTypeNum="0"
name="PTT.BK"
serviceId="1"
serviceName="ELEKTRON_DD"
Payload dataType="FieldList"
FieldList
FieldEntry fid="6" name="TRDPRC_1" dataType="Real" value="1.01"
FieldEntry fid="22" name="BID" dataType="Real" value="1.00"
FieldEntry fid="25" name="ASK" dataType="Real" value="203.04"
FieldEntry fid="32" name="ACVOL_1" dataType="Real" value="100000.00"
FieldEntry fid="11" name="NETCHNG_1" dataType="Real" value="203.15"
FieldEntry fid="267" name="ASK_TIME" dataType="Time" value="12:52:44:000:000:000"
FieldEntry fid="14305" name="EVENT_T_NS" dataType="Time" value="12:14:07:966:111:999"
FieldListEnd
PayloadEnd
UpdateMsgEnd
…
…
Summary
This article demonstrates how to set up a simulated Real-Time Optimized to test Refinitiv Real-Time SDK’s applications. It depends on the OpenSSL toolkit, node.js, and RSSL interactive provider. The first step is running the OpenSSL command to generate a certificate file and a private key file. These files are used to run the node.js and RSSL interactive provider applications. The next step is running the node.js application to handle REST services on the Refinitiv Data Platform including the RDP authentication service and streaming service endpoint information. The last step is running an RSSL interactive provider to provide real-time financial data. An RSSL interactive provider could be an ETA interactive provider, encrypted ADS, or testsever tool. Finally, it shows how to configure and run EMA C++ and Java examples on a Windows machine to connect to this simulated Real-Time Optimized environment.
References
1. Refinitiv Data Platform APIs (no date) Refinitiv Data Platform APIs | Refinitiv Developers. Available at: https://developers.refinitiv.com/en/api-catalog/refinitiv-data-platform/refinitiv-data-platform-apis (Accessed: October 21, 2022).
2. Refinitiv real-time C++ SDK (no date) Refinitiv Real-Time C++ SDK | Refinitiv Developers. Available at: https://developers.refinitiv.com/en/api-catalog/refinitiv-real-time-opnsrc/rt-sdk-cc (Accessed: October 21, 2022).
3. Refinitiv real-time Java SDK (no date) Refinitiv Real-Time Java SDK | Refinitiv Developers. Available at: https://developers.refinitiv.com/en/api-catalog/refinitiv-real-time-opnsrc/rt-sdk-java (Accessed: October 21, 2022).
