Whilst, Refinitiv offers several current APIs and SDKs for working with Real-Time streaming data - many of our users have a considerable deployment of applications developed using our legacy APIs.
For many if not most of these applications, the original developers have moved on and the maintenance of these applications is the responsibility of someone who may have never coded with the underlying API or SDK.
For these applications, there will come a time, where the current maintainers will be asked to rewrite/migrate these legacy API applications to a current Refinitiv API. Based on my experience of working with our users, this is often a cause of considerable anxiety for the developer - particularly if they have never used the legacy APIs and are suddenly faced with the task of learning two APIs - the legacy one and the newer alternative.
However, I have usually found the developer does not need to learn the legacy API in much detail. When helping developers in this situation, I find the following is usually sufficient to enable them to understand the legacy application code enough to migrate it to the newer API.
- Initially, a high-level overview outlining the key features of the legacy API helps the developer put things into context and gain a reasonable understanding of what the application is doing.
- I then follow this up with a walkthrough of key code snippets of an example application or their own codebase - highlighting key items such as configuration, initialisation, item subscription and so on.
In this document, I will attempt to document step 2 of the above. Whilst much of this information can be gleaned from the API's existing documentation, the information below would typically be spread out across several sections and/or documents.
If you have not already done so, I would recommend reading the RFA Overview document - a link to which is in the right-hand sidebar.
RFA to EMA Migration Notes
Key Functionality – Refer to the Notes section for details/context:
A high-level view of key features and functionality support across RFA and EMA
Feature/Functionality |
Enterprise Message API |
RFA |
Performance |
High |
High |
Data Formats |
OMM/RWF – Binary |
OMM/RWF – Binary |
Languages |
Java, C++ |
Java, C++ |
Native Cloud support |
Yes |
No |
Ease of Use implementation |
Yes |
No |
Event-driven |
Yes |
Yes |
Message-Based |
Yes |
Yes |
Configuration |
Programmatic + File-based |
Programmatic + File (+ Win Registry) |
TimeSeries |
No |
Yes (decoder utility) |
Full Depth Order Book |
Yes |
Yes (OMM Only) |
Chain handling |
*Helper Class |
Helper Class |
Snapshot |
Yes |
Yes |
Page based data |
Yes |
Yes |
Custom Domain Models |
Yes |
Yes (OMM Only) |
Views (Field Filtering) |
Yes |
Yes (OMM Only) |
Contributions to RTDS |
Yes |
Yes |
Contribution to Cloud |
Yes |
No |
Thread Safe and Aware |
Yes |
Yes |
Multi Consumer support |
Yes |
Yes |
Publisher/Provider |
Interactive + Non-Interactive |
Interactive + Non-Interactive / |
TIC/TIB/SASS support |
No |
Yes (MF only) |
ANSI Page support |
Yes |
Yes |
Notes:
Older versions of RFA support the Legacy string-based MarketFeed data format. More recent versions only support OMM (binary encoded) data format.
Performance – EMA offers a considerable improvement in performance over RFA MarketFeed due to optimised API design and the use of Binary Encoded OMM data.
For existing RFA OMM users, whilst performance figures are dependent on various factors such as watchlist size, volatility of instruments, networking etc, we can provide the following general guidance based on our internal testing and customer experiences:
· EMA C++ achieves similar performance to RFA OMM
· EMA Java has been shown to achieve higher performance than RFA Java
We hope to release Performance test figures in Q4 of 2021 which will provide more detail.
Data Format – Open Message Model uses the highly optimised binary encoded Refinitiv Wire format and allows richer Data model representation e.g. Full Depth Order book, Yield Curves, and other custom data domains.
Encoding and Decoding of OMM data is much simpler in EMA than RFA (example code snippet below)
Native Cloud Support – EMA can consume data from Cloud or deployed infrastructure. Easily switch between deployed and cloud data source (typically config change / few lines of code)
EMA is implemented as an Ease of Use Layer – much easier to learn and code with - requiring the developer to write a fraction of the code to achieve the same functionality as an RFA based application. This in turn makes the long-term maintenance of EMA based code much easier. See below for sample EMA code snippets that illustrate this Ease of Use.
Whilst EMA and RFA use Messages for data requests, responses, and contributions – constructing and parsing the message and payloads is much simpler in EMA than RFA.
Configuration – EMA defaults most Config parameters to (documented) common values which can then be optionally overridden programmatically. A config file containing multiple named configuration sets (e.g. Dev, QA, Production) can also be used - with the ability to specify a default config set. The developer can choose a non-default config by specifying the name of the required config set.
Whilst EMA does not provide a TimeSeries decoder, Refinitiv offers alternative Historical data sets which can be accessed via an easy to use RESTful interface.
Chain handling –a Chain Helper class for EMA Java is available on GitHub.
Custom Domain Models – using OMM allows the publication and consumption of rich custom data formats within your organisation
Field filtering is possible using the Views feature of EMA and RFA OMM – to optimise bandwidth usage and decoding of the payload.
Contribution to Cloud - EMA allows the contribution of data to the cloud-based Refinitiv Contribution Channel – eliminating the need for deployed contribution servers/gateways.
EMA and RFA allow a single application to consume data from multiple servers + sources and to horizontally scale the processing of larger watchlists on multi-core CPUs.
RFA vs EMA Configuration comparison
We have made considerable improvements in the configuration implementation in EMA
· RFA has specific objects for each element configured - EMA utilises a map structure to specify all configuration values
· RFA config is coupled tightly with Session and Connection in a very specific hierarchy/tree structure
· EMA’s programmatic config uses the same map layout as file-based config - making it easier for users to start developing with file-based and replace with programmatic as required
For a beginner’s guide to EMA Configuration please refer to Enterprise Message API (EMA) - Configuration Overview on our Developer Portal
Examples of code comparison
In the following section I will cover a couple of common coding scenarios – comparing RFA to EMA.
RFA C++ - Initialisation
For an RFA C++ consumer application as part of the initialisation process, you need to specify the minimum config values e.g.:
\Adapters\SSLED_Adapter\masterFidFile = "./appendix_a"
\Adapters\SSLED_Adapter\enumTypeFile = "./enumtype.def"
\Adapters\SSLED_Adapter\downloadDataDic = false
\Connections\Connection_SSLED\PortNumber = 8101
\Connections\Connection_SSLED\UserName = "<username>"
\Connections\Connection_SSLED\ServerList = "<serverList>"
\Connections\Connection_SSLED\connectionType = "SSLED"
\Sessions\Session1\connectionList = "Connection_SSLED"
In your application code, you then need to typically
· acquire/create a ConfigDataBase and StagingConfigDatabase
· load the config file (or registry) into the StagingConfigDatabase
· merge the StagingConfigDatabase into the ConfigDataBase
· create an Event Queue
· initialize the RFA context
· acquire a session (e.g. Session1 above)
· create a (legacy) MarketDataSubscriber or OMMConsumer
· create an event handler/callback client for the above subscriber or consumer instance
· register the above client with the above subscriber or consumer
· for OMM consumers:
send a Login Request and handle the response
load local or request download dictionary
handle the dictionary response
· for MD subscriber load local or request download dictionary
The above requires a minimum of 100+ lines of code - excluding the event handler/callback client code.
EMA C++ initialisation
At its most basic, for an EMA C++ consumer example the initialisation process is as follows:
AppClient client;
OmmConsumer consumer( OmmConsumerConfig().host( "ads1:14002" ).username( "user" ) );
The above two lines of code pretty much replace all the above RFA configuration file requirements + code (and more) - the 1st line creates an instance of the callback client, and the 2nd line results in the following actions:
· Create an OmmConsumer instance (our main subscription interface with EMA)
· if a config file does NOT exist, create a config using default values (or load the default config settings from the config file if one exists)
· override the default server details with ‘ads1’ on port 14002
· attempt a connection to the server
· send a login request for ‘user’ to the server
· on a successful login response, send a source directory request to the server
· request + download dictionary from the server (or load local if configured in the config file
For further details on EMA Configuration please refer to Enterprise Message API (EMA) - Configuration Overview on the Refinitiv Developer Portal
RFA Java vs EMA Java initialisation
Although class names/methods may vary somewhat, RFA Java requires similar initialisation steps as RFA C++ with some minor differences.
EMA Java initialisation is very similar to EMA C++:
AppClient appClient = new AppClient();
OmmConsumerConfig config = EmaFactory.createOmmConsumerConfig();
consumer = EmaFactory.createOmmConsumer(config.host("ads1:14002")
.username("user"));
As you will note, the key difference is the use of the EmaFactory to create the OMMConsumerConfig and OmmConsumer instances.
RFA Java - Subscribe to an instrument
For RFA, the subscription mechanism varies somewhat between the Legacy MarketData interface and the OMM interface.
Legacy MD interface code looks something like this:
MarketDataItemSub marketDataItemSub = new MarketDataItemSub();
marketDataItemSub.setItemName(itemName);
marketDataItemSub.setServiceName(_serviceName);
marketDataItemSub.setRequestedQualityOfService(qosr);
_marketDataSubscriber.subscribe(_eventQueue, marketDataItemSub, myClient);
In the above code, we
· create a MarketDataItemSub
· set the service name
· set item name (RIC code)
· subscribe via the MarketDataSubscriber instance and pass in the reference to the callback client (both of which were created earlier in the initialisation step)
RFA OMM interface code would look something like:
OMMItemIntSpec ommItemIntSpec = new OMMItemIntSpec();
OMMPool pool = OMMPool.create();
OMMMsg ommmsg = pool.acquireMsg();
ommmsg.setMsgType(OMMMsg.MsgType.REQUEST);
ommmsg.setMsgModelType(RDMMsgTypes.MARKET_PRICE);
ommmsg.setIndicationFlags(OMMMsg.Indication.REFRESH|OMMMsg.Indication.ATTRIB_INFO_IN_UPDATES);
ommmsg.setAttribInfo(serviceName, itemName, RDMInstrument.NameType.RIC);
ommItemIntSpec.setMsg(ommmsg);
itemHandle = _ommConsumer().registerClient(_eventQueue(), ommItemIntSpec, myClient)
In the above code we:
· create an OMM Item Interest Specification
· create a pool
· acquire an Omm Msg
· set the Msg Model Type (Level1 MarketPrice data)
· set the Msg Type to Request
· set flags to indicate a Streaming request
· set service name and RIC code
· assign the Msg to the Item Interest Specification
· finally, send our subscription request (i.e. register the interest specification with the previously created OMMConsumer)
EMA Java - Subscribe to an instrument
The above RFA Code code can be implemented with the following two lines of EMA Code:
ReqMsg reqMsg = EmaFactory.createReqMsg();
consumer.registerClient(reqMsg.serviceName("ELEKTRON_DD").name("IBM.N"), appClient);
We create a Request Msg, set the service name and RIC Code and then register the request with our previously created OmmConsumer interface and pass in a reference to our previously created callback client.
RFA C++ vs EMA C++ - Subscribe to an instrument
Although class names/methods may vary slightly, RFA C++ requires very similar steps as RFA Java (for both the legacy MarketData and OMM interfaces).
EMA C++ instrument subscription is again very similar to EMA Java:
consumer.registerClient( ReqMsg().serviceName( "ELEKTRON_DD" ).name( "BT.L" ), client );
RFA C++ - Extract, Decode and Display the data payload
Using the RFA Legacy MD interface, extracting and decode a typical MarketPrice payload requires several steps:
· the previously defined callback client’s processEvent() , receives the incoming Event
· examine the event type to confirm if it is a MarketDataItemEvent
· extract the MarketDataMsgType from the MarketDataItemEvent
· identify if it a status or data type Msg
· if the data buffer is empty extract & display the status information
· otherwise, unpack the buffer using a TibMsg helper function
· iterate through the list of fields, checking each field is valid and convert each field for display (or data processing)
The above involves typically 30+ lines of code.
For the RFA OMM interface, the process is something like this:
· the previously defined callback client’s processEvent() , receives the incoming Event
· examine the event type to confirm if it is an OMMItemEvent
· if so, extract the OMM Msg from the OMMItemEvent
· extract the OMM MsgType from the Msg
· confirm it is a RespMsg type and cast the Msg to a RespMsg
· if the RespMsg contains Status information extract and display it
· check the RespMsg ModelType to confirm it’s a data (e.g. MarketPrice) response
· extract the RespMsg Payload
· check the PayLoad Data type – for a Level 1 MarketPrice it would be a FieldList
· iterate through the FieldList, extracting each FieldEntry
· decode each FieldEntry
· extract each field’s definition to identify the field type
· decode / convert each field’s data value based on the field type
The above involves typically 100+ lines of code.
EMA C++ - Extract, Decode and Display the data payload
The above RFA C++ functionality can be implemented with a few lines of code EMA code, e.g. at its simplest:
void AppClient::onRefreshMsg( const RefreshMsg& refreshMsg, const OmmConsumerEvent& )
{
cout << endl << "Item State: " << refreshMsg.getState().toString() << endl;
if ( DataType::FieldListEnum == refreshMsg.getPayload().getDataType() )
decode( refreshMsg.getPayload().getFieldList() );
}
void AppClient::onUpdateMsg( const UpdateMsg& updateMsg, const OmmConsumerEvent& )
{
if ( DataType::FieldListEnum == updateMsg.getPayload().getDataType() )
decode( updateMsg.getPayload().getFieldList() );
}
void AppClient::decode( const FieldList& fl )
{
while ( fl.forth() )
cout << "Fid: " << fl.getEntry().getFieldId() << " Name: " <<
fl.getEntry().getName() << " value: " << fl.getEntry().getLoad().toString() << endl;
}
RFA Java vs EMA Java - - Extract, Decode and Display the data payload
Although class names / methods may vary slightly, RFA Java requires similar steps as RFA C++ (for both the legacy MarketData and OMM interfaces).
EMA Java extraction, decoding and displaying payload is again very similar to EMA C++:
public void onRefreshMsg(RefreshMsg refreshMsg, OmmConsumerEvent event)
{
System.out.println("Item State: " + refreshMsg.state());
if (DataType.DataTypes.FIELD_LIST == refreshMsg.payload().dataType())
decode(refreshMsg.payload().fieldList());
}
public void onUpdateMsg(UpdateMsg updateMsg, OmmConsumerEvent event)
{…}
void decode(FieldList fieldList)
{
Iterator<FieldEntry> iter = fieldList.iterator();
FieldEntry fieldEntry;
while (iter.hasNext())
{
fieldEntry = iter.next();
System.out.println("Fid: " + fieldEntry.fieldId() + " Name: " +
fieldEntry.name() + " value: " + fieldEntry.load());
}
}
As you will have noted, the EMA equivalent of RFA Consumer code is much simpler to implement, requiring far less learning time and coding effort.
Posting / Inserting data using RFA and EMA
Other than subscribing to / consuming data, the other most common use for RFA and EMA is to contribute (sometimes mistakenly referred to as publish) data – either to an internal data source or externally to Refinitiv or other vendors.
From an API perspective the action required for contributing data is referred to as sending an Insert (Legacy MD terminology) or a Post (RFA OMM/EMA terminology).
Posting / Inserting using RFA
Implementing basic Insert/Post functionality in RFA requires, pretty much the same level of complexity as with an RFA Consumer application – whether you are using the Legacy MD interface or OMM interface. Looking at some of the older RFA Insert/Post tutorials in Java and C++, the source files contain several hundred lines of code.
Posting using EMA
The initialisation and configuration steps for Posting with EMA are almost identical to the Consumer example shown earlier e.g. for EMA Java:
AppClient appClient = new AppClient();
consumer = EmaFactory.createOmmConsumer(EmaFactory.createOmmConsumerConfig()
.host("ads1:14002").username("user"));
ReqMsg reqMsg = EmaFactory.createReqMsg();
consumer.registerClient(reqMsg.domainType( EmaRdm.MMT_LOGIN ), appClient, consumer);
The key difference from the Consumer code is that we are registering the callback client for the Login request, so we can post the data for all items on the Login stream – rather than having to establish a stream for each item, we wish to Post data for.
In the RefreshMsg callback handler we then check for a valid Login response and post the data using the Login stream:
public void onRefreshMsg(RefreshMsg refreshMsg, OmmConsumerEvent event)
{
if ( refreshMsg.domainType() == EmaRdm.MMT_LOGIN &&
refreshMsg.state().streamState() == OmmState.StreamState.OPEN &&
refreshMsg.state().dataState() == OmmState.DataState.OK )
{
PostMsg postMsg = EmaFactory.createPostMsg();
RefreshMsg nestedRefreshMsg = EmaFactory.createRefreshMsg();
FieldList nestedFieldList = EmaFactory.createFieldList();
nestedFieldList.add(EmaFactory.createFieldEntry().real(22, 34, OmmReal.MagnitudeType.EXPONENT_POS_1));
nestedFieldList.add(EmaFactory.createFieldEntry().real(25, 35, OmmReal.MagnitudeType.EXPONENT_POS_1));
nestedFieldList.add(EmaFactory.createFieldEntry().time(18, 11, 29, 30));
nestedFieldList.add(EmaFactory.createFieldEntry().enumValue(37, 3));
nestedRefreshMsg.payload(nestedFieldList ).complete(true);
((OmmConsumer)event.closure()).submit(
postMsg.postId( postId++).serviceName( "NI_PUB" ).name( "TEST" )
.solicitAck( true ).complete(true)
.payload(nestedRefreshMsg), event.handle() );
}
}
The above snippet posts data to 2 numeric fields, a date field and an enumerated field. Note that we are using the (stream) handle from the Login event to Submit the message.
We also need to provide an additional callback to handle any acknowledgement message received from the server:
public void onAckMsg(AckMsg ackMsg, OmmConsumerEvent event)
{
System.out.println("Received AckMsg. Handle: " + event.handle() + " Closure: " + event.closure())
}
As you will note, the code required to Post data using EMA is relatively straightforward – whereas RFA requires considerably more code regardless of whether we are using the legacy MarketFeed or the newer OMM interface.
The above method of Posting data on the Login stream is referred to as Off-Stream posting. If you need to consume the instruments you are posting to, EMA and RFA also offer On-Stream posting. The key difference being that with On-Stream you request each instrument and use the (stream) handle for each instrument to submit the Post.
EMA code demonstrating both On-Stream and Off-Stream Posting are included in the 300_Series folder of the Training examples provided with the EMA C++ and Java SDK packages.
Publishing data using EMA C++ / Java
As with Posting data, publishing data using EMA is also much simpler than with RFA – our most basic RFA Provider example consists of over a thousand lines of code. Our most basic EMA Non-interactive provider example is implemented in less than 50 lines code and the Interactive Provider example in less than 100 lines of code (figures are for C++ versions – Java examples are slightly longer).
Closing Comments
Whilst the idea of migrating an application from RFA to EMA may initially seem daunting; the ease of learning, understanding, and coding of EMA – as illustrated above, should help allay much of the concern.
In addition to this, for most client applications I have come across – the classes used for consuming/posting/publishing Market Data are abstracted or isolated from the business logic classes which form the core of the implementation. Therefore, it should be possible to replace the consumer classes using a newer API, without affecting the rest of the code base.
If you have not already done so, I recommend you read in conjunction the linked Articles (see the upper right-hand side).
