Tutorial purpose

The aim of this tutorial is to explain the concepts related to data item publication and to teach you how to use EMA to publish a refresh message for a market price instrument.

To this aim we will go through the following sections:

• Introduction
• Background information about LSEG real-time infrastructures
• The MarketPrice Domain Model
• The Refresh message
• The State
• The Payload
• Putting it all together in the NiProvider
• The main workflow
• Build and run the application

Introduction

Now that we know how to properly connect a NI Provider application to a LSEG Real-Time Distribution System, it is time to get to the meat and actually publish data for the consuming applications. In this tutorial, you will learn how to use the EMA library to encode and publish data for a MarketPrice item.

Background information about LSEG Real-Time infrastructures

The aim of this section is to give you a brief overview of the LSEG Real-Time infrastructures, concepts, and workflows. For further details please consult the LSEG Real-Time API Concepts Guide and the EMA C++ RDM Usage Guide.

Distribution systems

The LSEG Real-Time Distribution System is an efficient and reliable distributed systems that enable the exchange of data items between Provider and Consumer applications. Before you start learning how to publish data, it is important you become familiar with the concepts described below.

Services

The system distributes data items via services. Each service provides a universe of data items (a.k.a instruments) and domain models that are served by provider applications. Several providers can serve the same service and a provider can serve several services.

A service is said Up when at least one of its providers is running and connected to the system. If no provider is available, the service is said Down and none of its data items can be consumed.  

Different services may deliver different data and different domain models for the same item. For example, on The RTDS Platform ELEKTRON_DD may provide market price level1 data for MSFT.O while DF_NAS_L2 provides market by order level 2 data for the same item.

Domain Models

Services and providers deliver data items for one or several domain models (a.k.a capabilities). Some of these models are defined by LSEG. They are then called Domain Models (RDM) - Market price, order book, and yield curve are well known Domain Models. Other domain models can be defined by users. They are then called User Domain Models (UDM).

Data items

Data items distributed on the system are identified by symbols. LSEG publishers use the RIC symbol type (Instrument Code) to identify the data items they publish. However, as the RTSDK is symbol type agnostic, Provider may use other types of symbols like ISIN, CUSIP, or SEDOL.

A Data item may be distributed by several services and provided for different domain models. For this reason, a symbol alone is not enough to identify a data item. It must be completed by the service name and the domain model.

These three elements (Service / Symbol / Domain Model) uniquely identify the data items distributed by the infrastructure. In this tutorial, we will publish Market Price data for the RIC SHARE-0 via the service NI-PUB.  

Note: The naming of the RIC with the  -0 (zero) suffix is intentional as in later tutorials we will demonstrate multiple items to publish, i.e. SHARE-0, SHARE-1, etc.

Non-Interactive Providers

NI Provider applications publish data items that are cached by the infrastructure and then distributed to the requesting consumers. Ni Providers can publish several data items of several domain models for one or several services. For this reason, they must indicate the service, the RIC, and the domain model of the item they publish.

Consumers

When a consuming application requests a data item, it must specify the three elements described above (Service, RIC, Domain Model). In return, and if the item is available, a refresh message (a.k.a image) is sent back by the system. In the case of market price data items, the refresh message contains the state and the complete data (all the fields) of the item. If the request indicated an interest in streaming data, the consumer may also receive subsequent updates and statuses. 

Provider / Consumer message flow

The diagram below illustrates an example of a message flow between a consumer that requested a streaming MarketPrice data item published by a NI Provider. This provider is connected to a remote source system that provides the actual data.

1. At start-up, the NI Provider connects to its source system and publishes the initial Refresh message for the item. This message contains all the item's fields. This data is cached by the LSEG Real-Time Distribution System but not forwarded to anyone as no consumer application registered an interest for this item yet.
2. Because the source system changed the item data, the NI Provider publishes an Update message that contains the fields that changed. The data cached by the distribution system is then updated with the new fields' values.
3. A consumer application that is interested in this item sends a streaming request for it. The item is identified by the Service, the RIC, and the Domain Model.
4. The distribution system sends back a Refresh message based on the data it cached for the item.
5. Because the source data changed again, the provider sends another Update message. The distribution system updates its internal cache.
6. As a consumer application is interested in this data item, the distribution system forwards the update to this consumer.
7. Because it detected a connection failure with its source system, the provider sends a Status message indicating that the data is suspect. Then it starts its failover process. The distribution system updates its internal cache with the new status.
8. The distribution system forwards the suspect status to the consumer.
9. After a while, the provider recovers the connection to its source system. Then, it publishes a Refresh message. This message contains all the item's fields and a status that indicates the data is Ok. The distribution system uses this message to refresh its internal cache.
   Note: Sending an unsolicited Refresh message after a connection recovery (either with the LSEG Real-Time Distribution System or with the source of data) is a good practice that providers are expected to follow. This is an excellent way to ensure the data cache is in sync with your data source.
10. The distribution system forwards the Refresh message to the consumer.
11. Because the source data changed, the provider sends an Update message. The distribution system updates its internal cache.
12. The distribution system forwards the update to the consumer.

The MarketPrice Domain Model

The MarketPrice domain allows us to publish Level I market information such as trades, indicative quotes, and top-of-book quotes. All information is sent as a list of field/value pairs that contain the information related to the data item (i.e. net change, bid, ask, volume, high, low, or last price). Each field is identified by a unique id and has an associated value of a predefined data type. The exhaustive list of fields with their ids, names, and data types are defined by a field dictionary held by the infrastructure. 

The diagram below illustrates the structure of a Market Price message. The FieldList conveys several field entries. Each FieldEntry holds a FieldId (a.k.a FID) and the current value of the related field. Details like the field name and the value type are defined in the distribution systems field dictionary. In this example, the first FieldEntry is for the DSPLY_NAME field and contains a 16-character long string.

Notes:

• If your provider application requires field dictionaries (because it relies on field names instead of field ids, for example) you must ask your administrator to provide you with the dictionaries used by your LSEG Real-Time Distribution System. They are provided as text files that your application will have to parse in order to build its in-memory representation of the field dictionaries. You will have to take care that the files used by your application are always in sync with the dictionaries used by your distribution system. 
• Examples of field dictionaries can be found in the RTSDK package (e.g. Ema\Etc\RDMFieldDictionary) or in the EMA C++ GitHub repository.

For MarketPrice data items, NI Providers can publish Refresh messages, Update messages, and Status messages.

For more details about the MarketPrice model and the related messages, please consult the Real-Time API Concepts Guide and the EMA C++ RDM Usage Guide.

The Refresh Message

For our first publication, we will send a Refresh message for the SHARE-0 market price item on the NI_PUB service. In order to tell the infrastructure what data item this message is published for, we will use the following information:

• Service name: “NI_PUB”. This is the default service name used by EMA for NI Providers. We will see in the next tutorial how to configure EMA in order to use another service name. But for now, we will stick with this one. If you try another service name you will get an exception error when you try to send the message.
• RIC: “SHARE-0”. This is an arbitrary name that we chose for this tutorial. You can change this name if you wish.
• Domain Type: MMT_MARKET_PRICE. This is the domain type to use for market price data.

The message will also transport the item state, and the item payload made of a field list.

To create a refresh message using EMA, you just have to instantiate a RefreshMsg object and initialize it by calling its different methods. Then, when you finished setting the different parts of the message, you must call the complete() method to indicate that the message is complete. For example, you could do something like:

The State

The state of the item is also set in the refresh message. This is done by calling the state() method like this:

This method gets 4 parameters:

• The stream state (e.g. Opened, Closed...).
• The data item state (e.g. Ok, suspect…).
• The status code (e.g. Not Found, Not Authorized, Timeout…).
• The status explanation.

Please consult the EMA C++ Reference Manual and the RDM Usage Guide for more details about the market price state.

The Payload

The refresh message will transport a payload made of the following field list:

The payload is simply added to the refresh message by calling the payload() method. But before you do that you must build the market price payload itself. To this aim, you must instantiate a FieldList object and initialize it by adding the fields and their values one by one. The FieldList class provides one method per field type. For example, to add a string field you must call the addAscii() method, to add a real field you must call the addReal() method, etc. Then, when the field list is complete, you must call the complete() method. Here is how you could proceed for the field list described above:

The FieldList class provides a number of addXXX() methods that require different parameters.

Sending the Refresh message

Once the refresh message is built and completed, it is time to publish it. This is done by calling the submit() method of the provider like this:

The first parameter is the refresh message to publish. The second parameter is a handle that uniquely identifies the item stream. This number is chosen by the provider application and will be used for every subsequent message sent for this item stream. Let us say you want to publish an update for the same data item, then you must submit the update message using the exact same handle (1 in this case).

Putting it all together in the NiProvider

Now that we saw how to build the payload, the state, and how to send a refresh message for a data item, we will put all this together in a new refresh() method of the NiProvider class.

This method takes two arguments, the service name and the item name which is the actual RIC. It is a public method of the NiProvider class, that we will call in our main workflow.

First of all, we test if the provider is connected and exit the method if it's not.

Then, we build and publish the message. As you can see, we leveraged the chained method call facility provided by EMA. This allows us to write a more concise and readable code. The code is indented in a way that you can easily tell what methods are called on what object. For example, from the indentation you can tell that the serviceName(), name(), domainType(), state(), payload() and the second complete() methods have been called on the RefreshMsg object that we created on the stack. From the indentation, you can also tell that the submit() method has two parameters: the RefreshMsg and the itemHandle.

Also to be mentioned:

• The item handle and field values are hardcoded. That is ok for now but will become an issue when we publish multiple items in later tutorials.
• The fields are built using the numerical field ids. This is not easy to read and not good programming practice.

We chose to implement this tutorial this way for the sake of simplicity, and because we are publishing only one data item. We will improve this code in the next tutorials.

The main workflow

The main workflow leverages the new refresh() method of the NiProvider class to implement the following:

• It creates the provider and connects it to the ADH
• It waits for 5 seconds to make sure the provider is actually connected to the ADH before sending the Refresh message.
  Note: This wait is a bit disproportionate as the provider generally connects almost instantly to the ADH. If for any reason the refresh() method was called before the connection, it would detect it and print an Error message on the console.   
• It uses the refresh() method to send a Refresh message for the NI_PUB/SHARE-0 market price item. 
• Once the message is published, the application waits for 60 seconds before exiting, so that you have time to subscribe to the item using the consumer application of your choice.

Build and run the application

Build the application and start it. Please refer to the Build and Run section within the first tutorial of this series (A barebones EMA NIP application shell) for detailed instructions.

This is what you should get:

1. A console application should open and display something like:

During the 60 seconds wait, open a consuming application and subscribe to the SHARE-0 market price item of the NI_PUB service. After a short while, you should receive values for the 5 fields (DSPLY_NAME/3), OPEN_PRC/19), HST_CLOSE/21), BID/22) and ASK/25) published by the NI Provider.
As an example, this is a screenshot of the Eikon Quote object that we used to subscribe to NI_PUB/SHARE-0. In the Eikon configuration, the NI_PUB service has been associated to the 'N' key letter:

3. After 60 seconds, the NI_PUB service goes down and the application exits.

4.Press a key to close the console when you are prompted to.
 

Troubleshooting

Q: When I build the tutorial project, I get errors like:

A: The ElektronSDKInstallPath environment variable is not set or set to the wrong path. See Setup the development environment.
 

Q: The application is stuck after the "Connecting Provider to ADH…" message is displayed.
After a while the application displays an error like: 

A: Verify that the ADH of your LSEG Real-Time Distribution System is up and that you properly set the host parameter in the EmaConfig.xml file. 
You can also use the telnet command tool to verify that your NIP application machine can connect to the ADH (telnet <ADH host> <port>). If the telnet succeeds but you still can’t connect, verify that you don’t have any firewall blocking the messages sent/received by the application.  
Ultimately, ask your administrator to help you to investigate with monitoring tools like adhmon.
 

Q: The “Waiting for 5 seconds...” message and the “Provider is connected” message get mixed up when displayed in the console.
A: This is perfectly normal and actually caused by the EMA background thread that prints the “Provider is connected” message at the same time the main application thread prints the “Waiting for 30 seconds...” message. This can be fixed either by choosing a mono threaded application model (see the Message Processing - dispatch section of the Requesting MarketPrice data EMA Consumer tutorial) or by printing these messages atomically (use critical sections or a single cout << call to print the whole message).