ARTICLE

Coding For High Precision Time

David Thomas

Audience

Developers responsible for maintenance of application code where it makes use of RFA or UPA and will be potentially exposed to the new high precision time fields.

Scope

This article provides a synopsis for the coding aspects of high precision time fields. Including; what applications are vulnerable when confronted with the high precision time fields, what to expect when this occurs and what you can do to ensure an application continues to operate correctly.

The factors informing the decision to add high precision time fields into applications or avoid them altogether is outside the scope of this article.

For further information see MiFID II - Extending the precision of timestamps supported on Refintiv Real Time.

Introduction

Following on from the Webinar session Introduction to MiFID II for Developers (narrow by 'Compiled' to find the video) I wanted to add some detail on the coding aspects of handling (or avoiding) the high precision time, not covered in the Webinar and include some information based on questions arising after the event.

For clarity, in the code extracts that follow, logging is used to illustrate the occurance and handling of the error or exception but in a production configuration this should be avoided when a time field raises the expected error or exception because the frequency with which this occurs will have an adverse effect on update processing performance.

Applications Considered

We will look at the following application types, if any action is required, where applicable what error is produced, with a brief description so that you can recognise when it has occurred in your application and what solutions are available;

Vulnerable Applications. Those at risk of failures when high precision time values are present in the data, including:
- OMM Applications Using UPA or RFA Version 7.x and Earlier Releases.
- 32-bit RFA OMM Applications.
Applications using a Non-OMM API. These applications will not encounter errors with the introduction of high precision time values on the data feed but the requirements of the application should be reviewed. These include:
- Applications Using RFA MarketData Interfaces.
- Applications Using Legacy RTDS (TREP) APIs.

Vulnerable Applications

Applications at risk of failures when high precision time values are present in the data.

TESTING NOTE: Failure will only occur when the microsecond or nanosecond components of the time value are populated, it is therefore essential that this is the case for any valid testing scenario in order to avoid the risk of a false positive test result.. e.g. Log the data being consumed by the application under test using a seperate application such as rmdstestclient that is capable of decoding high precision time values and check for the presence of micro-second or nano-second time values in the data during the testing period.

OMM Applications Using UPA or RFA Version 7.x and Earlier Releases

Existing OMM Applications using a version of RFA or UPA earlier than version 8.x will experience an error if it encounters a high precision time value during the decoding process. Applications that only use specific fields and do not attempt to decode any of the new high precision time fields (currently field names suffixed "_NS" by convention) will not encounter an error but in some cases applications which match this profile extract the value before determining if it is a required field and trigger an error as a result, therefore such applications must be thoroughly tested against data containing high precision time values.

RFA Java 7.x and Earlier Releases

When a Time field is encountered with either the microsecond or nanosecond components of the time value populated the API will raise an exception : com.reuters.rfa.omm.OMMException. with a message similar to this:

Exception in thread "main" com.reuters.rfa.omm.OMMException: Invalid size 8 for Time

The size reported is the encoded length determined by the presence of microsecond and nanosecond components.

Triggered in method:

    	
            com.reuters.rfa.omm.OMMData com.reuters.rfa.omm.OMMFieldEntry.getData(short type)

e.g.

    	
            // OMMData data 
// OMMFieldEntry fe 
// com.reuters.rfa.dictionary.FidDef fiddef 
data = fe.getData(fiddef.getOMMType());

Solutions Available

For applications using RFA Java 7.x and Earlier Releases, the solutions available are as follows:

Consider upgrading to one of the compatible API releases listed below.
Implement Pre-Decoding Checks.
Implement Exception Handling.

Upgrading the API and Pre-decoding check for high precision time values are mutually exclusive. Implementing Exception Handling is compatible and recommended for both Pre-Decoding checks and upgrading the API.

RFA C++ 7.x and Earlier Releases

When a Time field is encountered with either the microsecond or nanosecond components of the time value populated the API will raise an exception : InvalidUsageException with a message:

"Data decoding failed in DataBuffer::getTime(); Reason: RSSL_RET_INCOMPLETE_DATA".

Triggered in method:

    	
            const rfa::data::Time& rfa::data::DataBuffer::getTime () const;

But any method that implcitly requires access to the field data can raise this same exception and message, e.g.

    	
            bool rfa::data::DataBuffer::isBlank() const;

Solutions Available

The solutions available for applications using RFA C++ 7.x and Earlier Releases are:

Consider upgrading to one of the compatible API releases listed below.
Implement Pre-Decoding Checks.
Implement Exception Handling.

UPA Java 7.x and Earlier Releases

NOTE: This product will be obsolete after July 31st, 2021. Customers must upgrade to the Enterprise Transport API. For more information please see the Product Change Notification

UPA C 7.x and Earlier Releases

NOTE: This product will be obsolete after July 31st, 2021. Customers must upgrade to the Enterprise Transport API. For more information please see the Product Change Notification

32-bit RFA OMM Applications

Any existing 32-bit RFA OMM applications will be using a release earlier than version 8.x and will fail if it attempts to decode a high precision time value as described for RFA C++ 7.x and Earlier Releases.

Solutions Available

Solutions available for 32-bit RFA OMM Applications are:

Consider upgrading to one of the compatible API releases listed below: For those 32-bit RFA OMM applications that cannot be migrated to 64 bit at this time, the RFA 8.1.0.L1 release is supplied with 32-bit libraries, upgrading to this release will allow your application to operate when high precision time values are present in the data - but make sure your target OS and compiler are supported. If the application cannot be upgraded then the solutions available are the same as those given for 64-bit applications, i.e.
Implement Exception Handling.
Implement Pre-Decoding Checks.

Upgrading the API and Pre-decoding checks for high precision time values are mutually exclusive. Implementing Exception Handling is compatible and recommended for both Pre-Decoding checks and upgrading the API.

Applications Using Non-OMM APIs

An Application using a Non-OMM API will not be able to access the high precision time values; These new Time fields will yield a blank or be truncated to seconds, depending on the infrastructure version in use. Any such application should be subject to a requirements review to ensure that the regulatory changes do not neccesitate access to the high precision time values.

Applications Using RFA MarketData Interfaces

Because existing RFA MarketData applications will not have access to the high precision time values it will therefore not encounter an error when these are introduced into the instruments.

Applications using the RFA Market Data interface cannot be upgraded to version 8 because this interface has been removed from the release since version 8. If access to the high precision time fields is required it must be migrated to use the OMM interfaces using one of the compatible API releases.

Identifying the use of Legacy RFA MarketData Interfaces

Applications using legacy RFA Market Data interfaces can be identified by the use of any interface or class name starting 'MarketData' such as MarketDataSubscriber, but also the use of any interfaces with a name starting 'Tib' such as TibMsg or TibField. These applications can also be identified by their API configuration settings; The 'connectionType' parameter will be using a setting of "SASS3", "SSL" or ""SSLED" depending on the legacy protocol and RFA edition employed.

Applications Using Legacy RTDS (TREP) APIs

Legacy RTDS(TREP) APIs include SFC (Java, C++, or COM) or SSL (C), for more information see "API Catalog - Older APIs". No error will be encountered when the high precision time values are introduced to the instruments but if such an application requires access to them it must be migrated to one of the compatible API releases. Given that the RFA is now feature-complete (i.e. no further development), it is recommended that a strategic API is used e.g. Enterprise Messaging API (EMA). More information is available on the Developer Portal - see links below for Java and C++ versions.

Solutions

A Summary of the following potential solutions with example code extracts where applicable;

Upgrade API
Implement Exception Handling
Implement Pre-Decoding Check
Post-Decode Check

Solution: Upgrade API

Supported Operating Systems and Compilers

When considering an upgrade to any of the APIs listed below please ensure your target Operating System and compiler versions are supported, the latest information is listed in the API Compatibility Matrix, this is updated as and when support is added or removed. The release note also contains this information at the time of the release.

There are links to the Developer Web Portal in the tables below, where the example versions are given, each has a link to the appropriate release note, there is also a link to the API Compatibility Matrix associated with the API name.

APIs Compatible With High Precision Time Fields

The APIs that can already decode the new high precision time fields are;

All APIs bundled with the Real-Time SDKs i.e.

API	Language	Version
Enterprise Messaging API (EMA)	C++	Ideally rebranded versions e.g. Real-Time-SDK-2.0.1.L2 or later, see the download page
Enterprise Transport API (ETA)	C	Ideally rebranded versions e.g. Real-Time-SDK-2.0.1.L2 or later, see the download page
Enterprise Messaging API (EMA)	Java	Ideally rebranded versions e.g. Real-Time-SDK-2.0.1.L1 or later, see the download page
Enterprise Transport API (ETA)	Java	Ideally rebranded versions e.g. Real-Time-SDK-2.0.1.L1 or later, see the download page

2. All UPA 8.x releases

NOTE: UPA will be obsolete after July 31st, 2021. Customers must upgrade to the Enterprise Transport API (see above).
For more information please see the Product Change Notification

API	Language	Version
UPA	C	upgrade to ETA
UPA	Java	upgrade to ETA

3. All RFA 8.x releases Please Note that the RFA is now feature-complete (i.e. no further development), it is recommended that a strategic API is used e.g. Enterprise Messaging API (EMA) unless the application is already implemented with an existing version of RFA OMM Interface.

API	Language	Version
RFA	C++	Ideally rebranded versions e.g. rfa8.2.0.L2 or later, see the download page
RFA	Java	Ideally rebranded versions e.g. rfaj8.2.1.L1 or later, see the download page

Solution: Exception Handling

Applicable to RFA.

Although we are looking at Exception Handling specifically for defensive coding in the presence of high-precision timestamps for RFA versions earlier than 8.x, please take note of the Technical Bulletin "RFA AND EXCEPTION HANDLING" which advises its more general use.

All the exception handling examples are specifically designed to allow the application to continue processing the remaining data in the Field List after encountering a high precision time value, for performance reasons a more general exception handling policy would protect the application from complete failure but would not allow the remaining data to be processed. Such a policy would be implemented by placement of try-catch statements so that it is executed only once during the event processing method, e.g. on entry to the processEvent() method.

NOTE: If the presence of high precision time values are not an exception within the data-set on which the application will operate follow the advice in the Technical Bulletin "RFA AND EXCEPTION HANDLING" , as a more general approach to making your application more robust but to avoid frequently triggering exceptions use Pre-Decoding Checks.

Advantages

A general solution that allows an application to continue operating when confronted with decoding exceptions including, but not limited to, any high precision time or date-time values.
When an application is upgraded to version 8.x or later, the new fields will automatically become accessible, the application exception handling code remains valid and in place without change - it is simply no longer triggered by the new time fields because the new API version can decode them.

Disadvantages

The expense of setting up exception handling within update handlers is less attractive for applications with high performance requirements.
Triggering and handling exceptions is expensive in terms of performance cost, see exception handling note above.
When processing Time fields the application cannot determine the difference between encountering a high precision time value and a genuine Exception.

Implementation : RFA Java

To illustrate where to catch com.reuters.rfa.omm.OMMException when decoding a high precision timestamp value we will use RFA Java version rfaj7.6.1 and the example com.reuters.rfa.example.omm.cons.StarterConsumer.java from that Devkit.

File: GenericOMMParser.java

Method:

    	
            private static final void parseEntry(OMMEntry entry, PrintStream ps, int tabLevel)

    	
                { // Start of method body.
    try // try-catch to defend against decoding errors on new _NS Time fields.
        { // Original code starting with switch statement...
            switch (entry.getType())

    	
             default:
 dumpEntryHeader(entry, ps, tabLevel);
 parseData(entry.getData(), ps, tabLevel);
 break;
    } //End of switch statement with no alterations.
 } // catch to defend against decoding errors on new _NS Time fields. 
 catch (OMMException e)
 {
 ps.println("ERROR Invalid data: " + e.getMessage());
 }

Placement of the try-catch block within the parseEntry() method allows the application to continue processing the remaining fields in the FieldList after encountering a high precision time value.

Implementation : RFA C++

To illustrate where to catch InvalidUsageException when decoding a high precision timestamp value we will use RFA C++ version rfa7.6.2.L1.win-shared.rrg and the StarterConsumer example from that Devkit.

File: ..\StarterCommon\Decoder.cpp Method:

    	
            void Decoder::decodeFieldEntry( const FieldEntry& input )

    	
             if(_fd)
 fprintf(_fd, "\t");
 //Exception Handling code starts here;
 // enclose decodeData call in try block if it is a time field.
 if( _pCurrentFidDef->getOMMType() == rfa::rdm::RDMFidDef::RWF_TYPE_TIME  )
 {
 try {
 decodeData( input.getData(static_cast<UInt8>(_pCurrentFidDef->getOMMType())) );
 } // Add catch statement for the expected InvalidUsageException
 catch( InvalidUsageException& iue )
 {
 AppUtil::log(__LINE__, AppUtil::WARN, 
                         "Decoder::decodeFieldEntry() - Error decoding OMMType %d: %s\n"
 ,_pCurrentFidDef->getOMMType(), iue.getStatus().getStatusText().c_str());
 } 
 }
 else //Exception Handling code ends here.
 decodeData( input.getData(static_cast<UInt8>(_pCurrentFidDef->getOMMType())) );

Solution: Pre-Decoding Check

Applicable to: RFA versions earlier than 8.x. All editions.

Test the encoded length of the incoming time field. This will allow an application to identify if it exceeds the prescribed encoded-length limit for the API version employed and thereby avoid decoding that field. Thus the application continues to operate.

Advantages

More efficient than try-catch exception handling constructs.

Disadvantages

Requires more code effort in comparison with exception handling. All of the code points that explicitly decode the time value and those that implicitly access the time value (e.g. DataBuffer::isBlank()) must be changed.
In testing terms more effort is required because all the code-paths must be exercised in order to ensure that the defensive code is complete and correct.
This approach is designed to handle a specific problem, other issues will still throw an exception e.g. an invalid time range.
When the application is upgraded all the pre-decode checks must be removed before it can access the new time fields.

Implementation : RFA Java

To illustrate how to check for a high precision timestamp value before it triggers an exception we will use RFA Java version rfaj7.6.1 and the example com.reuters.rfa.example.omm.cons.StarterConsumer.java from that Devkit.

File: GenericOMMParser.java

Method:

    	
            private static final void parseEntry(OMMEntry entry, PrintStream ps, int tabLevel)

    	
             if (fe.getDataType() == OMMTypes.UNKNOWN)
 { // New code starts here
   //Test the data-type is a TIME field and its length exceeds the maximum for this RFA version.
 if(fiddef.getOMMType()==OMMTypes.TIME && fe.getData().getEncodedLength()>5)
 {
 System.out.println("FID "+fe.getFieldId()+" Invalid size "+
 fe.getData().getEncodedLength()+" for TIME.");
 return; // Do not decode this data.
 } // else it is safe to decode as before...
 data = fe.getData(fiddef.getOMMType());
 } // New code ends here
 else
 // defined data already has type

Implementation: RFA C++

Using RFA C++ release rfa7.6.2.L1.win-shared.rrg and example StarterConsumer from that DevKit to illustrate checking for a high precision timestamp value before it triggers an exception. File: ..\StarterCommon\Decoder.cpp Method:

    	
            void Decoder::decodeFieldEntry( const FieldEntry& input )

    	
            if(_fd)
 fprintf(_fd, "\t");
 // Pre-Decode Check Starts here...
 // Get the Data from the FieldEntry
 const rfa::common::Data& data = input.getData(
                                    static_cast<UInt8>(_pCurrentFidDef->getOMMType()));
 // If the data-type is a TIME field and its length exceeds the maximum for this RFA version.
 if( _pCurrentFidDef->getOMMType() == rfa::rdm::RDMFidDef::RWF_TYPE_TIME 
 && data.getEncodedBuffer().size() > 5 )
 {
 // then do not decode this field
 AppUtil::log(__LINE__, AppUtil::WARN, 
                     "Decoder::decodeFieldEntry() - Invalid data, OMMType %d: length %d\n"
 ,_pCurrentFidDef->getOMMType(), data.getEncodedBuffer().size());
 }
 else // Pre-Decode Check Ends here...
 // it is safe to decode as before...
 decodeData( input.getData(static_cast<UInt8>(_pCurrentFidDef->getOMMType())) );

    	
            protected int decodeFieldEntry(FieldEntry fEntry, DecodeIterator dIter, DataDictionary dictionary, StringBuilder fieldValue)

Line 710 :

    	
            case DataTypes.TIME:
 if( fEntry.encodedData().length()>5)
 {
 //Pre-decode check
 System.out.println(
     "Pre-decode check: aborted Time.decode(); Length= "+
 fEntry.encodedData().length());
 return CodecReturnCodes.SUCCESS;
 }
    ret = fidTimeValue.decode(dIter);

Solution: Post-Decode Check

Applicable to: UPA versions earlier than 8.x. All editions.

NOTE: This product will be obsolete after July 31st, 2021. Customers must upgrade to the Enterprise Transport API. For more information please see the Product Change Notification

Using A Custom Dictionary

Applicable to: RFA and EMA.

There is an approach to avoid decoding high precision time fields which requires maintainance of a separate data dictionary that does not contain the new high precision time fields, particularly for applications that cannot upgrade.

It is not suggested as a solution in this article but it is given some attention here because of reported issues in the field; You may consider it because it appears to offer a solution with the possibility of no code change or upgrade required in the application, if you are considering this tactic please note the following caveats:

Caveats

Some editions of RFA used exceptions when a field it attempted to decode was not present in the data dictionary. RFA C++ and Java editions will return null, in either case ensure your application manages this correctly. e,g, null pointer or unhandled exceptions.
Must ensure that the performance profile of the application is not adversely affected by the additional processing introduced by handling the missing FIDs.
Because of item 1. and 2. above this approach can only be used to protect the application where the presence of high precision time fields are an exceptional circumstance and not part of the data-set on which the application is expected to operate.
Thorough testing required - there is no rollback.
Short term solution. The technology allows any TIME data type to deliver high precision time, this strategy relies on adherence to the convention that Refinitiv will only publish high precision time values on fields suffixed '_NS'. Consequently any changes to this convention will require changes to the application and/or the dictionary.
The introduction of any new '_NS' fields must be removed from updates applied to the custom dictionary.
This creates potentially significant additional infrastructure setup and operational maintenance overheads to manage two distinct dictionaries including how they are delivered to the application.
Other issues will still throw an exception e.g. an invalid time range.

Decoding High Precision Time Fields

The two examples below illustrate the extraction of all the components of a high precision time value, the extracts of the resulting outputs are from a payload containing the following fields.

Sample Payload

Fid	Name	Value
2	RDNDISPLAY	116
6	TRDPRC_1	14.57
14263	ASK_TIM_NS	(blank data)
14264	BID_TIM_NS	13:02:35.997.0.0
14265	QUOTIM_NS	13:02:35.997.998.999
19	OPEN_PRC	84.94
4	RDN_EXCHID	202

Note:

The time field "ASK_TIM_NS" is encoded as blank,
"BID_TIM_NS" is populated up to millisecond (the microsecond and nanosecond components are blank) and
"QUOTIM_NS" is fully populated including microseconds and nanoseconds.

RFA Java Example

RFA Java rfaj8.0.1.L1.all.rrg

Using DevKit Example: com.reuters.rfa.example.quickstart.QuickStartConsumer

File: GenericOMMParser.java

Method:

    	
            public static final void parseData(OMMData data, PrintStream ps, int tabLevel)

Line: 561

    	
            try { 
       // ADDED: To show access to the microsecond and nanosecond components 
       // of a high precision timestamp 
       if( data.getType() == OMMTypes.TIME ) 
       { 
         OMMDateTime dtd = (OMMDateTime)data; 
         ps.println(String.format(" TIME: %02d:%02d:%02d.%d.%d.%d ", 
                 dtd.getHour(), dtd.getMinute(), dtd.getSecond(), dtd.getMillisecond(), 
                 dtd.getMicrosecond(), dtd.getNanosecond())); 
       } 
       else ps.println(data); 
    } 
    catch (Exception e) 
    { 
       byte[] rawdata = data.getBytes(); 
       ps.println(HexDump.hexDump(rawdata));
    }

Output:

...

FIELD_ENTRY 14263/ASK_TIM_NS:

FIELD_ENTRY 14264/BID_TIM_NS: TIME: 13:02:35.997.0.0

FIELD_ENTRY 14265/QUOTIM_NS: TIME: 13:02:35.997.998.999

...

EMA C++ Example.

Using EMA C++ Example: 120__MarketPrice__FieldListWalk

Contained in directory: <RT-SDK>\Ema\Examples\Training\Consumer\100_Series_Examples\

File: Consumer.cpp

    	
            void AppClient::decode( const FieldList& fl ) 
{ 
   while (fl.forth()) 
   { 
      // Code added to show extraction of individual time components starts here 
      const refinitiv::ema::access::FieldEntry& fe = fl.getEntry(); 
      // Check the FieldEntry is a Time value and not blank - extracting a time value 
      // when FieldEntry::getCode() returns Data::BlankEnum is an error and will 
      // raise OmmInvalidUsageException. 
      if (fe.getCode() != Data::BlankEnum && fe.getLoadType() == DataType::TimeEnum) 
      { 
         const OmmTime& ot = fe.getTime(); 
         // Access the data as OmmTime. 
         cout << "Fid: " << fe.getFieldId() << " Name: " << fe.getName() << " OmmTime value: " 
              << (int)ot.getHour() << ":" << setfill('0') << setw(2) << (int)ot.getMinute()<< ":" 
              << setfill('0') << setw(2) << (int)ot.getSecond() << "." << (int)ot.getMillisecond() 
              << "." << (int)ot.getMicrosecond() << "." << (int)ot.getNanosecond() << endl; 
      } 
      else // Code added to show extraction of individual time components ends here 
           cout << "Fid: " << fl.getEntry().getFieldId() << " Name: " << fl.getEntry().getName() 
                << " value: " << fl.getEntry().getLoad().toString() << endl; 
   } 
}

Output:

...

Fid: 14263 Name: ASK_TIM_NS value: (blank data)

Fid: 14264 Name: BID_TIM_NS OmmTime value: 13:02:35.997.0.0

Fid: 14265 Name: QUOTIM_NS OmmTime value: 13:02:35.997.998.999

...

Glossary

Term	Description
RFA	Robust Foundation API. It Provides a single, low-level, thread-safe API that applications can use with RTDS to publish and consume data. This API is now feature-complete in terms of its development lifecycle. It consisted of an OMM interface and a legacy MarketData interface.
UPA	Ultra Performance API. This is a transport layer API which has now been incorporated into the strategic Real-Time SDK as the Enterprise Transport API (ETA).
OMM	Open Message Model
RTDS	Real-Time Distribution System (formerly known as Thomson Reuters Enterprise Platform for Real-Time - TREP)
EMA	Enterprise Messaging API. A strategic API available in C++ and Java which is part of the API suite provided with the Real-Time SDK.
ETA	Enterprise Transport API. A strategic API available in C and Java which is part of the API suite provided with the Real-Time SDK.
SSL	Sink Source Library. A legacy 'C' language Marketfeed API.
SFC	System (formerly SSL) Foundation Classes. A legacy API release for C++, Java and COM.

References

MiFID II - Extending the precision of timestamps supported on Refinitiv Real Time

Webinar Recording: Introduction to MiFID II for Developers on YouTube

Technical Bulletin: IMPACT OF WIRE FORMAT CHANGES ON THE APIS

Technical Bulletin: RFA AND EXCEPTION HANDLING

The API Compatibility Matrix for the various APIs

API Catalog - Older APIs

Articles

Developer Tools

Use Cases

Upcoming Events

Past Events

About Us

Coding For High Precision Time

Contents

Audience

Scope

Introduction

Applications Considered

Vulnerable Applications

OMM Applications Using UPA or RFA Version 7.x and Earlier Releases

RFA Java 7.x and Earlier Releases

Solutions Available

RFA C++ 7.x and Earlier Releases

Solutions Available

UPA Java 7.x and Earlier Releases

UPA C 7.x and Earlier Releases

NOTE: This product will be obsolete after July 31st, 2021. Customers must upgrade to the Enterprise Transport API. For more information please see the Product Change Notification

32-bit RFA OMM Applications

Solutions Available

Applications Using Non-OMM APIs

Applications Using RFA MarketData Interfaces

Identifying the use of Legacy RFA MarketData Interfaces

Applications Using Legacy RTDS (TREP) APIs

Solutions

Solution: Upgrade API

Supported Operating Systems and Compilers

APIs Compatible With High Precision Time Fields

Solution: Exception Handling

Advantages

Disadvantages

Implementation : RFA Java

Implementation : RFA C++

Solution: Pre-Decoding Check

Advantages

Disadvantages

Implementation : RFA Java

Implementation: RFA C++

Solution: Post-Decode Check

Using A Custom Dictionary

Caveats

Decoding High Precision Time Fields

RFA Java Example

EMA C++ Example.

Glossary

References