.HSI

0005.HK

Considering raw data from RFA and EMA tracing log below, The garbage string from above sample texts usually contains three bytes character at the beginning of the text and there are 0x1B, 0x25 and 0x30 which is garbage characters "", "%" and "0".

.HSI

0005.HK

Actually, these three bytes character are not garbage character. There is an escape sequence used by our data feed component for the text encoded with UTF-8. You may also see difference format of garbage character at the beginning of the string and it could be other character sets used by our internal publisher application. If the application calls DataBuffer.getAsString() method in RFA or using getAscii in EMA, it would return the actual string including the escape sequence. This is because the method does not have a built-in RMTES decoder. Normally RFA application can use RMTES Converter interface in order to get an actual UTF-8 text from the DataBuffer or just calling fieldEntry.getRmtes().toString() in EMA. We will provide more details about the background of the escape sequence in the next section.

RMTES Encoding

Basically, RMTES uses ISO 2022 escape sequences to select the character sets used. RMTES provides support for Reuters Basic Character Set (RBCS), UTF-8, Japanese Latin and Katakana (JIS C 6220 - 1969), Japanese Kanji (JIS X 0208 - 1990), and Chinese National Standard (CNS 11643-1986). RMTES also supports sequences for character repetition and sequences for partial updates. These characters set or format internally used by various infra components. Unfortunately, there is no open RMTES encoder library provide for external user or customer. There are many questions from application developers who want to create publishing or contributing application, and it has to publish RMTES string which contains non-ASCII text such as Japanese and Chinese characters. Implementing their own RMTES encoder is not easy and it's a very complex format.

However, there is an alternative choice for this case as the developer can use the switching function provided for encoding RMTES string and switching from default ISO 2022 scheme to UTF-8 character set. It could say that the developer can use the UTF-8 character set to publish data for RMTES field type. They can use the function to encode the UTF-8 string into RMTES string and then publish the string to the system.

The switching function is permitted only at the very start of a field and it contains three bytes of characters that are 1B 25 30.

The function is permitted only at the very start of a field. The application could prepend 0x1B, 0x25, 0x30 to the UTF8 string and encode that way as an RMTES type. The escape sequence characters indicate to the RMTES parser or decoder that it’s supposed to be a UTF-8 string. As a result, there is a reason that the samples data from field DSPLY_NMLL for RIC ".HSI" and "0005.HK" start with the three bytes escape sequence character 0x1B 0x25 and 0x30.

Please note that you need to be very careful with using that three-byte string, as it can cause the UTF-8 string to be longer than the cached dictionary values which are the size of RWF LEN column in a byte from RDMFieldDictionary. It can cause display issues if they’re going through the infra.

The non-ASCII character such as Chinese, Thai, Japanese and Korea language can be used UTF-8 character set, therefore, the application can use this way to encode the non-ASCII text instead. We will talk about the implementation in RFA and EMA C++/.NET in the next section.

Publishing non-ASCII RMTES string in RFA application

As described earlier, the publishing or contributing application can encode the non-ASCII text with UTF-8 instead of implementing RMTES encoder. The application can just prepend the three bytes escape sequence 0x1B, 0x25, and 0x30 to the UTF8 string and encode in the DataBuffer class as RMTES string.

Below is sample codes for RFA C++ and you can add the codes to our provider example such as Provider_Interactive example provided with the RFA C++ package in order to test the codes. You can modify codes in file MarketByPriceStreamItem.cpp method MarketPriceStreamItem::encodeMarketPriceFieldList to publish additional FID which use RMTES_String data type. In this case, we will add FID 1352 DSPLY_NMLL which holds local language instrument name to the codes.

Below sample codes use u8 literal from C++ 11 to create UTF-8 string. You can use another library or a different approach to create a UTF-8 string. Note that u8 literal requires Visual Studio 2013 or later version and you may need to add -std=c++11 when compiling the example on Linux.

And the following codes are for RFA.NET Provider application. You can try sample C# codes with Provider_Interactive from RFA.NET package. Just modify the codes in file MarketPriceStreamItem.cs method EncodeMarketPriceFieldList.

Below is sample outgoing RSSL tracing log generated by the RFA Provider application. Comparing the data with sample data we described earlier, it's the same and identical one. 

An output from RFA StarterConsumer which has RMTES decoder, it can display the Chinese text correctly without garbage character. We will describe decoding RMTES string in Decoding RMTES section.

Publishing non-ASCII RMTES string in EMA application

Using the same approach in EMA C++ is easier than RFA. You can just add the UTF-8 string like the RFA C++ to FieldList and then publish the data to the wire. In order to test the codes, you can use Provider interactive example from EMA C++ package. For testing the codes, we use example 200MarketPriceStreaming to publish the same Chinese text. You can modify and use below codes in file IProvider.cpp method AppClient::processMarketPriceRequest.

The outgoing message from EMA tracing log shows the same value as RFA. You should get the same result as RFA C++ when using OMM Consumer which has RMTES decoder subscribe the data from the Provider example.

Decoding RMTES

RFA and EMA generally provided RMTES converter or parser interface for converting the encoded RMTES string payload received as part of the OMM data to a Unicode string. It helps display news in international languages with UCS2 format or transfer data through the network in ISO 2022 and UTF-8. The following section will provide a guideline for applications that want to display non-ASCII string correctly.

Displaying non-ASCII RMTES string in RFA Consumer application

RFA provides RMTESConverter interface to convert RMTES string back to a readable string. The interface is useful for converting the encoded RMTES string payload received as part of the OMM data to a Unicode string. It helps display news in international languages with the UCS2 format or transfer data through the network in ISO 2022 and UTF-8 format. The RMTESConverter interface provides the following major functions:

• setBuffer(): binds a buffer to a converter (with a heap allocation). If the offset passed into the function is -1, the buffer passed in contains full field data and RFA will internally cache the buffer in memory. If the offset is equal to or greater than 0, the buffer passed in contains partial field data and RFA will internally apply the partial field data on the earlier cached buffer based on offset. The cached buffer should contain full field data.

• getAsCharString(): returns a UTF-8-formatted string as an RFA_String.

• getAsShortString(): returns a UCS2-formated string as an RFA_Vector in RFA C++ or List in RFA.NET. Each item in the vector is a short which holds a char and each item in List is a char which holds a character. The size of the vector or List is the total number of characters.

Sample Codes for RFA C++

RFA C++ example such as StarterConsumer cannot decode data buffer which contains non-ASCII text. However, the user can add RMTESConverter to StarterConsumer when it decoding the data buffer. You can open StarterConsumer project and add below codes to StandardOut.cpp in method void StandardOut::out( const DataBuffer& dataBuffer ). Note that below codes are just sample codes for decoding generic RMTES string and it does not support partial RMTES update. If you want to decode partial RMTES update, an application has to cache the RMTESConverter object and apply all received changes to them. You can find more details about RMTESConverter from RFA C++ Development guide.

Open StarterConsumer_Log on a text editor, you should see chinese text displayed correctly. 

Sample Codes for RFA.NET

Like the RFA C++, we use StarterConsumer to demonstrate the codes for decoding the RMTES string and shows the non-ASCII text returned by DSPLY_NMLL fid. You can open StarterConsumer project from RFA.NET package and then open file StandOut.cs and then add below codes to method public void Out(DataBuffer dataBuffer)

If the application has to works with a partial RMTES update message, it has to catch the RMTESConverter object as explained in RFA C++. Please refer to the RFA.NET Development guide for more details about RMTES Usage.

Displaying non-ASCII RMTES string in EMA C++ Consumer application

Using RMTES on EMA C++ is easier than RFA C++/.NET. This is because EMA provides a built-in RMTES decoder. If needed, the application can cache RmtesBuffer objects and apply all received changes to them. Please refer to EMA C++ Documents for more information about RmtestBuffer class.

Sample Codes for EMA C++

Below are sample codes that the application can cache the RmtesBuffer object and apply all received changes to them.

In case that application wants to decode RMTES field and do not need to works with the partial update, it can just call thomsonreuters::ema::access::FieldEntry::getRmtes() to get RMTESBuffer object when it decoding the FieldEntry which has DataType as RmtesEnum.

Conclusions

In this article, we explain the details of how to decode and encode non-ASCII text in field type RMTES using EMA and RFA C++ or .NET edition. We also provide a snippet of codes to demonstrate the API usages. The developer can try the codes with the examples application provided in the RFA or EMA package.

References

For further details, please check out the following resources:

• RFA C++ Developer Guide
• RFA.NET Developer Guide
• EMA C++ Developer Guide