10 important things you need to know before you write a Refinitiv Real-Time application

1 - Refinitiv Real-Time services deliver a consolidated global data feed (skip this section)

Refinitiv Real-Time is a low-latency consolidated global data feed that can deliver full tick, depth-of-market data. This global consolidated feed connects your applications to thousands of exchanges and OTC-traded markets, and this coverage is growing all the time. The Refinitiv Real-Time service can be consumed either via a deployed Refinitiv Real-Time Distribution system (formerly known as TREP) or via the Cloud-based Refinitiv Real-Time Optimized service (RRTO). 

Using the deployed Distribution System it is possible to consume the data as a full tick, conflated and/or delayed service as per your needs and requirements. 

The Cloud-based RRTO service is currently delivered as a Bandwidth Optimized Network service - although there are discussions around providing a full tick service via the cloud in the future.

2 – Refinitiv Real-Time services are available via the Refinitiv Real-Time Distribution system as well as Cloud-Based Delivery (skip this section)

The Refinitiv Real-Time data can be delivered to your applications either via the cloud-based Refinitiv Real-Time Optimized service or via a deployed Refinitiv Real-Time Distribution Systems (RTDS). The RTDS gives you access to the Real-Time data feed and optionally to other vendors data feeds. You can even use RTDS to publish your own data and data models.

The delivery method is chosen (by market data teams generally) depending on criteria like:

• The use case: Do you want to consume or to publish data or both? Do you need fault tolerance? Do you want to be able to run several instances of your application (Prod, Backup and Dev)?
• The number of applications and users that connect to the platform
• Data quality you need (full tick , conflated and/or delayed)
• The available bandwidth to connect to the platform
• …

3 - The Refinitiv Real-Time Distribution System (RTDS) can support Publishers, Consumers and Hybrid applications (skip this section)

The purpose of the RTDS is not only to consume Refinitiv Real-Time data but optionally publish your own data for others to consume. Indeed, the RTDS components are highly efficient real time, streaming architectures and as such they are extensively used as middleware, enabling an ecosystem of financial applications. While providers implement services and expose a certain set of capabilities (e.g. content, workflow, etc.), consumers use these capabilities for their specific purposes (e.g. trading screen applications, black-box algorithmic trading applications, etc). In some cases, a single application can even function as both a consumer and a provider to leverage and add value to existing data (e.g., a computation engine, value-add server, etc.).

Under this approach, Refinitiv Real-Time and RTDS applications can be of one of the 3 following types:

• Consumer applications: Applications that subscribe to data items and/or contribute to data items available on the platform.
  Note: The concept of contribution is explained in the section below.
• Provider applications: Applications that publish data items and optionally receive contributions from consumer applications.
  Note: Provider applications can be either interactive or non interactive. Please consult the related tutorials to learn more on this subject.
• Hybrid applications: Applications that are able to consume data from the real time platform, transform it and republish it for other consuming applications.

4 - Refinitiv Real-Time applications use the Publish/Subscribe paradigm (skip this section)

The exchange of real time data between RTDS and Refinitiv Real-Time applications relies on the Publish/Subscribe paradigm. On Refinitiv Real-Time systems, this paradigm is completed by the concept of data contribution:

• Publication: Provider applications publish data items, aimed at consumer applications. Whenever a data item changes, the provider keeps consumer applications updated by publishing the new values on the real time platform.
• Subscription: Consumer applications subscribe to one or more data items published by one or more providers. When a consumer subscribes to a data item, a dedicated communication stream is opened between the platform and the application for this specific data item. Then, the platform uses this communication stream to keep the consumer application updated with the latest values published by the provider.
• Contribution/Posting: Optionally, a consumer application can issue a post message that contains a data contribution for a specific item published by a provider. When it receives the contribution, the provider has the option to either positively acknowledge it (ACK) or negatively acknowledge it (NAK). Then, depending on its internal logic, the provider can forward the contributed data to an external system, or re-publish it or use it for any other purpose.
  Note: Provider applications must be programmatically ready to receive contributions (Post messages). Not all of them are implemented to do so.

These 3 concepts are illustrated below. For the contribution part, the provider is represented re-publishing the data. This is just an example. As explained above, this behavior is totally dependent on the provider implementation.

Note: The Refinitiv Real-Time Distribution System provides additional and more complex means to exchange data between applications (e.g. private streams, generic messages…). These advanced concepts are not covered by this article. Please consult the Real-Time SDK documentation for more details on these topics.

5 – Data items are uniquely identified by 3 elements (Name, Type and Service)(skip this section)

A data item on the platform is the representation of a real world financial object like the quotation of a stock or its order book. Because these real world financial objects can be of different types (quotation, order book…), the data items representing them use different data structures and have different behaviors. For example:

• The quotation of the AIRBUS stock traded on Euronext Paris (Level 1 data) is represented by a data item called AIR.PA that is made of a list of fields. These fields contain the different values of the stock quotation (bid, ask…).
• The order book for the same AIRBUS stock (Level 2 data) is represented by another data item also called AIR.PA that is made of several lists of fields. Each list of fields representing a specific order of the book. The fields of one list hold the different values of an order (price, size, side…).

In order to identify data items, we use their names (e.g. AIR.PA). But as you can see in the above example, two data items representing different real world objects related to the same stock may have the same name. For this reason we need an additional element to distinguish data items on the platform. This second element is the data item type that can be, for example, MMT_MARKET_PRICE for a Level1 market price data or MMT_MARKET_BY_ORDER for a full order book. A third element is also required because data items with the same name and the same type can potentially be published by different services of the platform (a.k.a. sources). For example, your platform might provide multiple services publishing data from different vendors. This is the reason why the publishing service must also be indicated to completely identify a data item on the platform.

These three elements (Name/Type/Service) uniquely identify each data item of the real time platform. You need to indicate them whenever you subscribe to or publish any data item.

With the above example:

• The AIRBUS Level 1 market price would be identified by:
  • Name: AIR.PA
  • Type: MMT_MARKET_PRICE
  • Service: ELEKTRON_DD
• The AIRBUS Level 2 full order book would be identified by:
  • Name: AIR.PA
  • Type: MMT_MARKET_BY_ORDER
  • Service: ELEKTRON_DD

About data item names

The financial world uses different identification codes for naming financial instruments. Depending on the location of the exchange, one coding system or another is used. One can mention the CUSIP numbers used in the U.S and Canada, the SEDOL codes used in the United Kingdom or the Wertpapierkennnummer used in Germany. For obvious reasons, a consolidated global data feed like Refinitiv Real-Time needs a unified coding system that allows identifying financial instruments traded all over the world.

As a financial instrument may be traded on several stock exchanges (e.g. the Apple Inc. stock is traded on the Nasdaq, the New York Stock Exchange, the Xetra…) this coding system must be able to uniquely identify an instrument on a specific exchange. This is not a characteristic every coding system offers. For example the ISIN code, that is an international code, doesn’t distinguish instruments traded on several exchanges. Because of that, an ISIN code alone is not enough to identify an instrument. It must be completed by an exchange code to be accurate.

In order to solve the issues related to the global naming and the uniqueness of names, Reuters defined its own coding system based on RICs (Reuters Instrument Codes). This coding system is international and uniquely identifies instruments traded on different exchanges. For example: RACE.MI is the RIC referring to FERRARI being traded on the Milan Stock Exchange while RACE.N refers to the same stock trading on the New York Stock Exchange.

It is important to note that Refinitiv Real-Time platforms and their APIs are instrument code agnostic. This means that your applications (consumers and providers) can use the naming system they want (as long as they use the same). However, it is worth mentioning that RICs are largely used on Refinitiv Real-Time platforms. For example all Refinitiv Real-Time services use RICs.

About data item types

The domain type of a data item is a unique number that defines the data constructs, the semantics and the behaviors of the item. Because different data items related to the same financial instrument can use the same name, it is important to indicate the domain type when an application publishes or subscribes to an item. For example, the “6758.T” RIC associated with the MMT_MARKET_PRICE (6) domain type refers to the Level 1 “Sony Corp.” stock quotation, while the “6758.T” RIC with the MMT_MARKET_BY_ORDER (7) domain type refers to the Level 2 full order book of the same stock.

About services

Provider applications publish data items on RTDS via services. Each service has a unique name and a unique ID. A service can provide data items for one or several domain types. It is important to note that several services can provide data items with the same names and the same domain types but not necessarily with the same data. For this reason, data items with identical names and domain types but published via different services must be considered as different items. For example, two different providers could publish a MMT_MARKET_PRICE (6) data item with the same RIC but with values calculated by two different algorithms.

6 – You can choose from several different APIs to build your application (skip this section)

In order to publish or consume data from a Refinitiv Real-Time system your application needs to leverage one of the Refinitiv Real-Time APIs. Depending on the language and the technology you use to build it, several options are at your disposal.

Either you use the Real-Time SDK and its software libraries that expose the two following APIs:

• The Enterprise Transport API (ETA):
  This is a low level API available for Java and C/C++. It has been designed for applications demanding very high throughput & ultra low latency.
• The Enterprise Message API (EMA):
  This is a higher level API available for Java and C++. EMA is build on top of ETA and provides both ease of use & high performance.

Or you use the WebSocket API exposed by the platform:

• The WebSocket API for for Pricing Streaming and Real-Time Services:
  This API has the same level of abstraction than EMA but works with request/response JSON messages exchanged with the platform via WebSockets. Thanks to the WebSocket technology, this API can easily be integrated into a multitude of client technology environments such as scripting and web.
  Note: This API is available starting with RTDS 3.1. It doesn’t yet support the publication features.

Strategic APIs vs Legacy APIs

Developers who worked with previous Refinitiv real-time APIs may be interested in understanding what strategic API roughly corresponds to what legacy API. The following diagram gives an overview of these equivalences:

Notes:
- UPA (the Ultra Performance API) was renamed to ETA (the Enterprise Transport API), UPA and ETA are the same API.
- In this picture, SFC and the RFA legacy API are represented at the same level as EMA, but they do not provide the exact same set of features. The same comment applies to the SSL API, the RV API and ETA.

7 - Refinitiv Real-Time APIs are messaging oriented (skip this section)

Refinitiv Real-Time APIs are messaging oriented APIs that rely on data models and messages to receive and publish data items. In the Refinitiv Real-Time documentation you will often find references to these models, messages and formats. These are usually called by their acronyms: DMM, RDM, OMM and RWF. This section gives you an overview of these acronyms and the related concepts:

• Domain Message Models (DMM):
  They describe a specific arrangement of data constructs, semantics and behaviors that define real world data objects. For example: A specific DMM has been defined to describe Level I market prices instruments and their related behaviors. Three different DMMs have been defined to describe the different types of Level II data (Full order books, Market depth information, Market maker quotes & trade information). There are more DMMs, including one that describes yield curves.
  Even if a number of Domain Message Models already exist, new DMMs can be defined to represent other objects of your application domain. Each DMM has a unique Domain Type numeric value used to identify it.
  It is important to note that DMMs are not handled by any of our Real-Time APIs (with the exception of the “Login”, “Source Directory” and “Dictionary” models handled by EMA). Their data structures, semantics and behaviors are documented and must be managed at a higher level of the application. Our APIs just transport the Domain Type value so that the application knows what kind of data it receives and how to handle it.
• Refinitiv Domain Models (RDM):
  An RDM is a DMM defined by Refinitiv. The list of RDMs, their related data constructs, their semantics and their behaviors are defined in the “RDM Usage Guide” provided with the Real-Time SDK documentation. All RDMs have a Domain Type value less than 128.
• User Defined Domain Models:
  A User Defined Domain Model is a domain message model defined by a party other than Refinitiv. These may be defined to solve a specific user need which is not resolvable through the use of an RDM. Any user-defined model must have a domain type value between 128 and 255. If needed, domain model designers can work with Refinitiv to define their models as standard RDMs. This allows for the most seamless interoperability with future RDM definitions and with other Refinitiv  Real-Time products.

• Open Message Model (OMM):
  OMM is the collections of message headers and data constructs for representing data containers (FieldList, Map, Vector…) and primitive types. DMMs and RDMs rely on OMM to define their related data structures.
• Reuters Wire Format (RWF):
  RWF is the encoded representation of OMM. RWF is a highly-optimized, binary format designed to reduce the cost of data distribution as compared to legacy wire formats.

The Refinitiv WebSocket API uses a JSON representation of OMM with "display-ready" data values. These JSON messages transport the same DMMs and RDMs than the Real-Time SDK APIs. As for the Real-Time SDK applications, WebSocket applications must rely on the domain types to properly handle the messages they receive.

8 - Refinitiv Real-Time APIs are event-driven (skip this section)

Refinitiv Real-Time APIs are asynchronous and event-driven APIs that rely on the Publish/Subscribe paradigm. This means that you cannot get a data item via the returned value of a simple method call. With a Real-Time API, you must:

1. Call methods to subscribe to the data item you are interested in.
2. Wait for the API to call back asynchronously, the methods that you defined, with the values of the item you requested.

If you indicated an interest in updates, the API keeps you updated by calling your methods again with the latest values published for the data item.

Further below are two examples of this workflow, one for a Level 1 data item (e.g. a stock) and one for a Level 2 data item (e.g. a full order book). In order to understand these workflows, make sure you are familiar with the 5 following concepts.

Note: for the sake of simplicity and because it is the most widespread use case, the explanations and examples given in this section are from the perspective of a consumer application. The same concepts or sometimes the opposite concepts related to Provider and Hybrid applications can be deduced from these explanations.

Important concepts _{(Skip these concepts)}

Subscription

This is the action of indicating your interest in receiving the values of a specific data item. A subscription is done by calling one of the Real-Time APIs. For example, with the Enterprise Message API (EMA) you subscribe to a data item by calling the registerClient method of the OmmConsumer class. When you subscribe to a data item you expect the API to send you events and messages that contain the values and statuses of the data item you subscribed to.

If you are not interested anymore in receiving events for a data item, you can close the subscription (a.k.a. unsubscribe). With EMA this is done by calling the unregister method of the OmmConsumer class.

Data Item stream

The action of subscribing to a data item opens a stream by which you will receive the events and messages that contain the values and statuses of the subscribed item. With EMA this stream is identified by the numerical handle returned by the registerClient method when you subscribed.

Image

The image of a data item is the complete set of values that represents the item. This set of values is defined by the model type of the data item you subscribed to. Applications often need to keep a local (e.g. in-memory) representation of data items images. This cache of images is not managed by our Real-Time APIs but must be implemented by the application and kept up-to-date thanks to the messages it receives from the API. 

Images values come with the Refresh messages sent by the API. With EMA these messages are sent via the onRefreshMsg callback method of the OmmConsumerClient interface registered in the subscription phase. For Level 1 data items, Refresh messages contain the complete set of values of the image. On the contrary, for Level 2 data items Refresh messages may not be complete and only contain a part of the image. Because Level 2 images can be very large, they are transported in a sequence of Refresh and Update messages that must be consolidated by the application.

Updates

An update is the subset of values of a data item that recently changed. The update doesn’t contain the complete image but only the values that changed. Applying these values to the latest image of a data item allows keeping it up-to-date.

Updates are sent by the API via Update messages. With EMA these messages are sent via the onUpdateMsg callback method of the OmmConsumerClient interface registered at subscription time. Update messages always contain values for pieces of data that the application already received.

Status Event

Status Events indicate the state of a data item values or the state of the related event stream. Status are sent by the API via Status messages. With EMA these messages are sent via the onStatusMsg callback method of the OmmConsumerClient interface registered at subscription time.

Workflow examples

Level 1 Data Item example _{(Skip this workflow)}

This example illustrates the possible interactions between an application and the Enterprise Message API when subscribing to a Level 1 data item like a stock quotation. This includes the subscription, the reception of data (several images and updates) and statuses, the subscription closing.

•  
• Step 1: The application calls registerClient to subscribe to a Level 1 data item and to register a client callback object. This object will be used by the API to send messages to the application. Among other parameters, the application indicates the name of the item, the service it is published on and its model type (e.g. APPL.O, ELEKTRON_DD, MMT_MARKET_PRICE (6)). After, the registerClient method returned, the application waits for EMA to send messages related to the subscribed data item.
• Step 2: EMA sends a Refresh message to the application via the onRefreshMsg client callback. Since this Refresh message is for a Level 1 data item, it contains the complete list of fields of the item. The application can optionally cache this data creating a local representation of the data item image.
• Step 3 & 4: EMA sends several Update messages to the application via the onUpdateMsg client callback. Each Update message contains the subset of the data item fields that changed. If it preserved an image of the item, the application can update it with these new values.
• Step 5: EMA sends a Status message to the application via the onStatusMsg client callback. This message contains a Suspect data state, indicating that the data received for the item is not reliable anymore. One possible reason for this may be that the provider application has been disconnected from its source of data for example. Depending on their internal logic and purpose, applications behave differently when they receive this kind of status. For example, a calculation engine may decide to stop calculations and raise an alert while a Desktop application like Eikon may decide to continue displaying the data but with a particular background color (red) to inform the user that the displayed data is not reliable anymore.
• Step 6: EMA sends a new Refresh message with an Ok data state. This indicates that the issue that caused the Suspect data state is now solved and that the data is reliable again. The Refresh message also contains a fresh and complete list of fields that the application can use to refresh the image of the data item.
• Step 7, 8 & 9: EMA regularly sends Update messages to keep the application updated with the latest changes of the data item.
• Step 10: The application calls unregister to close the subscription. When calling unregister the application indicates the stream handle the registerClient method returned at Step1. After unregister returns, EMA will not send any new message to the application for this item stream.

Level 2 Data Item example _{(Skip this workflow)}

This example illustrates possible interactions between an application and the Enterprise Message API when subscribing to a Level 2 data item like an order book. This includes the subscription, the reception of data (the image via several messages and the updates) and the subscription closing.

• Step 1: The application calls registerClient to subscribe to a Level 2 data item and to register a client callback object. This object will be used by the API to send messages to the application. Among other parameters, the application indicates the name of the item, the service it is published on and its model type (e.g. APPL.O, ELEKTRON_DD, MMT_MARKET_BY_ORDER (7)). After, the registerClient method returned, the application waits for EMA to send messages related to the subscribed data item.
• Step 2: EMA sends a Refresh message to the application via the onRefreshMsg client callback. Since the Refresh message is for a Level 2 data item and because Level 2 data items images may be very large, this Refresh message may not contain the complete order book. This is the case here as the Refresh contains an indication that it is not complete. If the application wants to build a local representation of the order book’s image, it must cache the received orders and consolidate the image when it will receive the other orders via subsequent Refresh and Update messages.
• Step 3: EMA sends a second Refresh that contains additional orders. The application can use it to consolidate the order book’s image.
• Step 4: EMA sends an Update message that contains new values for already received orders. The application can use this message to consolidate the order book’s image. It is important to note that at this point the image of the order book is still not complete.
• Step 5: EMA sends a third Refresh message that is indicated as complete. This message contains the last orders needed to complete the image. The application can use them to consolidate the order book’s image. From this point the image can be considered as complete.
• Step 6, 7 & 8: EMA regularly sends Update messages to keep the application updated with the latest changes of the order book. These updates can be used to update the order book’s image.
• Step 9: The application calls unregister to close the subscription. When calling unregister the application indicates the stream handle the registerClient method returned at Step1. After unregister returns, EMA will not send any new message to the application for this item stream.

9 – Fields are described by dictionaries

Most of the items distributed on Refinitiv Real-Time platforms use the field list data structure. This data structure is made of a list of field/value pairs that transport several values of an item. Level 1 data items or individual orders of Level 2 order books are good examples of items made of field lists.

Each element of a field list has a unique field ID (FID) and a value. OMM messages that contain Field Lists only transport these two elements. Other information like the name of the field or the type is defined within a field dictionary that is either downloaded from the real time platform or provided as a file beside the application. Our Real-Time APIs extensively use this dictionary to decode the fields they receive from the platform.

Applications, on the other hand, do not always need to use this dictionary. Indeed, most of them know in advance the fields they need and their types. This allows them to hardcode this information within their source code. For example, an application that calculates the cross between two currencies could retrieve BID and ASK fields using hardcoded FIDs (22 and 25) or names (“BID” and “ASK”) and then directly try to retrieve their values as Real64 values. The application doesn’t need any dictionary for this. The field’s information can be hardcoded as it is very unlikely that the FIDs, names or types of BID and ASK change anytime soon. On the other hand, a more generic application that would display metadata about the fields it receives would need to retrieve this information from the field dictionary.

In any event, you need the field dictionary either in a human readable format – to find out as a developer what to hardcode in your application – or programmatically so that your application can access the fields metadata.

A human readable file version of the dictionary can be requested from the administrator of your real time platform. Alternatively you can use the dictionary examples provided with the Real-Time SDK, but it’s best to use to the dictionary of the platform. Dictionaries example files (“RDMFieldDictionary”) can be found in the “Cpp-c\etc” or the “Java\etc” folders of the Real-Time SDK (C++ or Java edition respectively).

Programmatically, field information is accessible via the DataDictionary object. With ETA, this object must be initialized with a dictionary loaded from a file or downloaded from the platform. With EMA, the DataDictionary object can be retrieved from a field list using the EMA DictionaryUtility class.

Note: The field dictionary is completed by another dictionary used for enumerated values. This dictionary called "enumtype.def" defines all possible values of every enumerated field. The file version of this dictionary is available alongside the field dictionary (“RDMFieldDictionary”). Programmatically, the DataDictionary object includes enumerated values information.

10 – You are not alone – The Force may not be with you, but the Refinitiv Developer Community is

If you are an experienced developer who has previously worked with legacy Refinitiv real-time APIs like RFA, you will probably find the move to EMA very easy. On the other hand, if you do not have any prior experience with this kind of APIs or if you start learning the lower level Enterprise Transport API (ETA), you may need help at some point.

If you find yourself in this situation, remember that you are not alone. A lively community of developers composed of Refinitiv employees and customers is here to help you. This community can be joined on the Refinitiv Developer Community Platform, via the Developer Portal. There you will find there articles, tutorials and plenty of other Refinitiv API related learning materials. You can also ask questions to the community and find the answers you need on the Discussion Forums. 

Don’t hesitate to join the Refinitiv Developer Community. It is in free access and open to everyone. Register here and get full access to the best learning content for Refinitiv APIs.

My other articles

If you liked this article, you may be interested in reading my other articles:

• Introduction to the "Refinitiv API Samples" Github group
• Simplifying Content Access in EMA
• The EMA ease of use in action (Video article)
• A simple MarketPrice object for EMA
• Simple Chain objects for EMA