Tutorial purpose

The aim of this tutorial is to explain the concepts related to data item publication and to teach you how to use EMA to publish a refresh message for a market price instrument.

To this aim we will go through the following sections:

• Introduction
• Background information about real-time infrastructures
• The MarketPrice Domain Model
• The Refresh message
• The State
• The Payload
• Putting it all together in the NiProvider
• The main workflow
• Build and run the application

Introduction

Now that we know how to properly connect a NI Provider application to a Real-Time Distribution System, it is time to get to the meat and actually publish data for the consuming applications. In this tutorial, you will learn how to use the EMA library to encode and publish data for a MarketPrice item.

Background information about the Real-Time infrastructures

The aim of this section is to give you a brief overview of real-time infrastructures, concepts, and workflows. For further details please consult the Real-Time SDK Concepts Guide and the EMA Java RDM Usage Guide.

Distribution systems

The real-time infrastructures like RTDS or Real-Time platform are efficient and reliable distributed systems that enable the exchange of data items between Provider and Consumer applications. Before you start learning how to publish data, it is important you become familiar with the concepts described below.

Services

The system distributes data items via services. Each service provides a universe of data items (a.k.a instruments) and domain models that are served by provider applications. Several providers can serve the same service and a provider can serve several services.

A service is said Up when at least one of its providers is running and connected to the system. If no provider is available, the service is said Down and none of its data items can be consumed.  

Different services may deliver different data and different domain models for the same item. For example, on the Enterprise Platform IDN_RDF may provide market price level1 data for MSFT.O while DF_NAS_L2 provides market by order level 2 data for the same item.

Domain Models

Services and providers deliver data items for one or several domain models (aka capabilities). Some of these models are defined by LSEG. They are then called Domain Models (RDM) - Market price, order book, and yield curve are well known Domain Models. Other domain models can be defined by users. They are then called User Domain Models (UDM).

Data items

Items distributed on the system are identified by a unique RIC. They may be distributed by several services and provided for different domain models. For this reason, a RIC alone is not enough to identify a data item. It must be completed by the service and the domain model.

These three elements (Service / RIC / Domain Model) uniquely identify the data items distributed by the infrastructure. In this tutorial, we will publish Market Price data for the RIC SHARE-0 via the service NI-PUB.  

Note: The naming of the RIC with the  -0 (zero) suffix is intentional as in later tutorials we will demonstrate multiple items to publish, i.e. SHARE-0, SHARE-1, etc.

Non-Interactive Providers

NI Provider applications publish data items that are cached by the infrastructure and then distributed to the requesting consumers. Ni Providers can publish several data items of several domain models for one or several services. For this reason, they must indicate the service, the RIC, and the domain model of the item they publish.

Consumers

When a consuming application requests a data item, it must specify the three elements described above (Service, RIC, Domain Model). In return, and if the item is available, a refresh message (a.k.a image) is sent back by the system. In the case of market price data items, the refresh message contains the state and the complete data (all the fields) of the item. If the request indicated an interest in streaming data, the consumer may also receive subsequent updates and statuses. 

Provider / Consumer message flow

The diagram below illustrates an example of a message flow between a consumer that requested a streaming MarketPrice data item published by a NI Provider. This provider is connected to a remote source system that provides the actual data.

1. At start-up, the NI Provider connects to its source system and publishes the initial Refresh message for the item. This message contains all the item's fields. This data is cached by the RTDS but not forwarded to anyone as no consumer application registered an interest for this item yet.
2. Because the source system changed the item data, the NI Provider publishes an Update message that contains the fields that changed. The data cached by the RTDS is then updated with the new fields' values.
3. A consumer application that is interested by this item sends a streaming request for it. The item is identified by the Service, the RIC and the Domain Model.
4. The RTDS sends back a Refresh message based on the data it cached for the item.
5. Because the source data changed again, the NI Provider sends another Update message. The RTDS updates its internal cache.
6. As a consumer application is interested by this data item, the RTDS forwards the update to this consumer.
7. If the NI Provider detected a connection failure with its source system, it sends a Status message indicating that the data is suspect. Then it starts its failover process. The RTDS updates its internal cache with the new status.
8. The RTDS forwards the suspect status to the consumer.
9. After a while the NI Provider recovers the connection to its source system. Then, it publishes a Refresh message. This message contains all the item's fields and a status that indicates the data is Ok. The RTDS uses this message to refresh its internal cache.
   Note: Sending an unsolicited Refresh message after a connection recovery (either with the RTDS or with the source of data) is a good practice that providers are expected to follow. This is an excellent way to ensure the RTDS data cache is in sync with your data source.
10. The RTDS forwards the Refresh message to the consumer.
11. Because the source data changed, the NI Provider sends an Update message. The RTDS updates its internal cache.
12. The RTDS forwards the update to the consumer.

The MarketPrice Domain Model

The MarketPrice domain allows to publish Level I market information such as trades, indicative quotes, and top-of-book quotes. All information is sent as a list of field/value pairs that contain the information related to the data item (i.e. net change, bid, ask, volume, high, low, or last price). Each field is identified by a unique id and has an associated value of a predefined data type. The exhaustive list of fields with their ids, names and data types are defined by a field dictionary held by the infrastructure. 

The diagram below illustrates the structure of a Market Price message. The FieldList conveys several field entries. Each FieldEntry holds a FieldId (a.k.a FID) and the current value of the related field. Details like the field name and the value type are defined in the field dictionary. In this example, the first FieldEntry is for the DSPLY_NAME field and contains a 16 characters long string.

Notes:

• If your provider application requires field dictionaries (because it relies on field names instead of field ids, for example) you must ask your RTDS administrator to provide you with the dictionaries used by your RTDS. They are provided as text files that your application will have to parse in order to build its in-memory representation of the field dictionaries. You will have to take care that the files used by your application are always in sync with the dictionaries used by your RTDS. 
• Examples of field dictionaries can be found in the Real-Time SDK package (e.g. Ema\Etc\RDMFieldDictionary) or in the EMA Java GitHub repository.

For MarketPrice data items, NI Providers can publish Refresh messages, Update messages, and Status messages.

For more details about the MarketPrice model and the related messages, please consult the Real-Time SDK Concepts Guide and the EMA Java RDM Usage Guide.

The Refresh message

For our first publication, we will send a Refresh message for the SHARE-0 market price item on the NI_PUB service. In order to tell the infrastructure what data item this message is published for, we will use the following information:

• Service name: “NI_PUB”. This is the default service name used by EMA for NI Providers. We will see in the next tutorial how to configure EMA in order to use another service name. But for now, we will stick with this one. If you try another service name you will get an exception error when you try to send the message.
• RIC: “SHARE-0”. This is an arbitrary name that we chose for this tutorial. You can change this name if you wish.
• Domain Type: MMT_MARKET_PRICE. This is the domain type to use for market price data.

The message will also transport the item state, and the item payload made of a field list.

To create a refresh message using EMA, you just have to create a RefreshMsg object and initialize it by calling its different methods. Then, when you finished setting the different parts of the message, you must call the complete() method to indicate that the message is complete. For example, you could do something like:

The State

The state of the item is also set in the refresh message. This is done by calling the state() method like this:

This method gets 4 parameters:

• The stream state (e.g. Opened, Closed...).
• The data item state (e.g. Ok, suspect…).
• The status code (e.g. Not Found, Not Authorized, Timeout…).
• The status explanation.

Please consult the EMA Java Reference Manual and the RDM Usage Guide for more details about the market price state.

The Payload

The refresh message will transport a payload made of the following field list:

The payload is simply added to the refresh message by calling the payload() method. But before you do that you must build the market price payload itself. To this aim, you must create a FieldList object and initialize it by adding the fields and their values one by one using field entries. The FieldEntries class provides one method per field type. For example, to set a string value in a FieldEntry you must call the ascii() method, to set a real value you must call the real() method, etc. Here is how you could proceed for the field list described above:

The FieldEntry class provides a number of methods to set different value types. These methods require different parameters. 

Sending the Refresh message

Once the refresh message is built and completed, it is time to publish it. This is done by calling the submit() method of the provider like this:

The first parameter is the refresh message to publish. The second parameter is a handle that uniquely identifies the item stream. This number is chosen by the provider application and will be used for every subsequent message sent for this item stream. Let us say you want to publish an update for the same data item, then you must submit the update message using the exact same handle (2 in this case).

Putting it all together in the NiProvider

Now that we saw how to build the payload, the state and how to send a refresh message for a data item, we will put all this together in a new refresh() method of the NiProvider class.

This method takes two arguments, the service name and the item name which is the actual RIC. It is a public method of the NiProvider class, that we will call in our main workflow.

First of all, we test if the provider is connected and exit the method if it's not.

Then, we build and publish the message. As you can see, we leveraged the chained method call facility provided by EMA. This allows us to write a more concise and readable code. The code is indented in a way that you can easily tell what methods are called on what object. For example, from the indentation you can tell that the serviceName(), name(), domainType(), state(), payload() and the complete() methods have been called on the RefreshMsg object that we created using the EmaFactory. From the indentation, you can also tell that the submit() method has two parameters: the RefreshMsg and the itemHandle.

Also to be mentioned:

• The item handle and the field value is hardcoded. That is ok for now but will become an issue when we publish multiple items in later tutorials.
• The fields are built using the numerical field ids. This is not easy to read and not good programming practice.

We chose to implement this tutorial this way for the sake of simplicity, and because we are publishing only one data item. We will improve this code in the next tutorials.

The main workflow

The main workflow leverages the new refresh() method of the NiProvider class to implement the following:

• It creates the provider and connects it to the ADH
• It waits for 5 seconds to make sure the provider is actually connected to the ADH before sending the Refresh message.
  Note: This wait is a bit disproportionate as the provider generally connects almost instantly to the ADH. If for any reason the refresh() method was called before the connection, it would detect it and print an Error message on the console.
• It uses the refresh() method to send a Refresh message for the NI_PUB/SHARE-0 market price item. 
• Once the message is published, the application waits for 60 seconds before exiting, so that you have time to subscribe to the item using the consumer application of your choice.

Build and run the application

Build the application and start it. Please refer to the Build and Run section within the first tutorial of this series (A barebones EMA NIP application shell) for detailed instructions.

This is what you should get:

1. The application should display something like:

2.  During the 60 seconds wait, open a RTDS consuming application and subscribe to the SHARE-0 market price item of the NI_PUB service. After a short while, you should receive values for the 5 fields (DSPLY_NAME/3), OPEN_PRC/19), HST_CLOSE/21), BID/22) and ASK/25) published by the NI Provider.

As an example, this is a screenshot of the Eikon Quote object that we used to subscribe to NI_PUB/SHARE-0. In the Eikon configuration, the NI_PUB service has been associated to the 'N' key letter:

3.  After 60 seconds, the NI_PUB service goes down and the application exits.

Troubleshooting

Q: When I build or run the tutorial, it fails with an error like:

A: The JAVA_HOME environment variable is not set, or set to the wrong path. See Setup the development environment section of the first tutorial.

Q: When I build the tutorial, I get ”<path>/javac: No such file or directory” or when I run the tutorial, I get  ”<path>/java: No such file or directory” error like

A: The JAVA_HOME environment variable is not set, or set to the wrong path. See Setup the development environment.

Q: When I build or run the tutorial, I get "Finding Jar files in <path>” and “The system cannot find the path specified.” errors like

A: The cause could be:

• The RTSDK_JAVA_HOME  environment variable is not set or set to the wrong path. See Setup the development environment.

Q: When I build or run the tutorial, I get "<path to /*jar>: No such file or directory” error like

A: The cause could be:

• The RTSDK_JAVA_HOME  environment variable is not set or set to the wrong path. See Setup the development environment.

Q: When I build the tutorial, I get "package ... does not exist" and "cannot find symbol" errors like:

A: The RTSDK_JAVA_HOME environment variable is not set, or set to the wrong path. See Setup the development environment section of the first tutorial.

Q: When I run the tutorial, I get a JNI error with a NoClassDefFoundError exception like:

A: The RTSDK_JAVA_HOME environment variable is not set, or set to the wrong path. See Setup the development environment section of the first tutorial.

Q: The application is stuck after the "Connecting Provider to ADH…" message is displayed.
After a while the application displays an error like: 

A: Verify that the ADH of your RTDS infrastructure is up and that you properly set the host parameter in the EmaConfig.xml file. 
You can also use the telnet command tool to verify that your NIP application machine can connect to the ADH (telnet <ADH host> <port>). If the telnet succeeds but you still can’t connect, verify that you don’t have any firewall blocking the messages sent/received by the application.  
Ultimately, ask your RTDS administrator to help you to investigate with RTDS monitoring tools like adhmon.

Q: The NI Provider application can connect and publish data to ADH, but the consumer application does not receive data and shows the following Status Message

Or

A: It means the published Service is not match with the NI Service defined in the ADH configurations. Please contact RTDS administrator to help you to check the NI Service name defined in your RTDS infrastructure.