Introduction

Update: March 2025

This article shows how developers may use the Real-Time Websocket API aka Websocket API application to subscribe Machine Readable News (MRN) from Real-Time Distribution System (Advanced Data Hub and Real-Time Advanced Distribution Server), then assemble and decode MRN textual news message to display news in a console/Jupyter Notebook.

Machine Readable News (MRN) is an advanced service for automating the consumption and systematic analysis of news. It delivers deep historical news archives, ultra-low latency structured news, and news analytics directly to your applications. This enables algorithms to exploit the power of news to seize opportunities, capitalize on market inefficiencies, and manage event risk.

Update (As of March 2025): The GitHub example 's Main/Master branch now supports the Real-Time - Optimized (RTO) connection. You do not need to switch to the ERT-in-Cloud branch anymore.

• The RTO with Version 2 Authentication console example: Please check the mrn_console_rto_v2.py example file.
• The RTO notebook example: mrn_notebook_app_rto_v1.ipynb notebook file.
• The deployed Real-Time Distribution System (RTDS) examples are mrn_console_rtds.py console application and mrn_notebook_app_rtds.ipynb notebook files.

WebSocket API Overview and Prerequisite

The WebSocket API is direct access that enables easy integration into a multitude of client technology environments such as scripting and web. This API runs directly on your Real-Time Distribution System infrastructure or the on the cloud and presents data in an open (JSON) readable format.

This example is focusing on the Machine Readable News (MRN) data processing only. I highly recommend you check the WebSocket API Tutorials page if you are not familiar with WebSocket API.

The Tutorials page provides a step-by-step guide (connect, log in, request data, parse data, etc) for developers who are interested in developing a WebSocket application to consume real-time data from Real-Time platform. The Tutorials also cover both Real-Time Optimized (RTO) and deployed Real-Time Distribution System (RTDS) infrastructure connection scenarios.

MRN Overview

MRN is published over Real-Time platform using an Open Message Model (OMM) envelope in News Text Analytics domain messages. The Real-time News content set is made available over MRN_STORY RIC. The content data is contained in a FRAGMENT field that has been compressed and potentially fragmented across multiple messages, to reduce bandwidth and message size.

A FRAGMENT field has a different data type based on a connection type:

• RSSL connection (RTSDK C++/Java/C#): BUFFER type
• WebSocket connection: Base64 ASCII string

The data goes through the following series of transformations:

1. The core content data is a UTF-8 JSON string
2. This JSON string is compressed using gzip
3. The compressed JSON is split into several fragments (BUFFER or Base64 ASCII string) which each fit into a single update message
4. The data fragments are added to an update message as the FRAGMENT field value in a FieldList envelope

Therefore, to parse the core content data, the application will need to reverse this process.

Data Model

Five fields, as well as the RIC itself, are necessary to determine whether the entire item has been received in its various fragments and how to concatenate the fragments to reconstruct the item:

• MRN_SRC: identifier of the scoring/processing system that published the FRAGMENT;
• GUID: globally unique identifier for the data item, all messages for this data item will have the same GUID value;
• FRAGMENT: compressed data item fragment;
• TOT_SIZE: total size in bytes of the fragmented data;
• FRAG_NUM: sequence number of fragments within a data item, which is set to 1 for the first fragment of each item published

and is incremented for each subsequent fragment for the same item. A single MRN data item publication is uniquely identified by the combination of RIC name, MRN_SRC, and GUID.

Fragmentation

For a given RIC/MRN_SRC/GUID combination, when a data item requires only a single message, then TOT_SIZE will equal the number of bytes length in the FRAGMENT and FRAG_NUM will be 1.

When multiple messages are required, then the data item can be deemed as fully received once the sum of the number of bytes length of each FRAGMENT equals TOT_SIZE. The consumer will also observe that all FRAG_NUM range from 1 to the number of the fragment, with no intermediate integers skipped. In other words, a data item transmitted over three messages will contain FRAG_NUM values of 1, 2, and 3.

The WebSocket application also needs to convert the FRAGMENT field data from Base64 string to bytes data before further process this field. Python application can use the base64 module to decode Base64 string to bytes data.

Compression

The FRAGMENT field is compressed with gzip compression, thus requiring the consumer to decompress to reveal the JSON plain-text data in that FID.

When an MRN data item is sent in multiple messages, all the messages must be received and their FRAGMENTs concatenated before being decompressed. In other words, the FRAGMENTs should not be decompressed independently of each other.

The decompressed output is encoded in UTF-8 and formatted as JSON. Python application can uses zlib module to decompress JSON string.