Exploring News Metadata - With Refinitiv Data Platform And Python

Article

Exploring News Metadata - with Refinitiv Data Platform and Python

August 12, 2020

Zoya Farberov

Developer Advocate

Introduction

Refinitiv Data Platform provides simple web-based API access to a broad range of content. Refinitiv Data Platform (RDP) News service encompasses diverse news content sets:

News Alerts
Headline Search
Story Retrieval
Metadata Retrieval

News metadata describes the content of a story and is created and packaged with the story. It is most often retrieved and presented with the story. The hierarchy of news metadata is available for separate retrieval. Metadata enables accurate and efficient search and discovery of the relevant news content.

Next, we are going to discuss RDP News metadata retrieval capabilities and use examples of code in Python.

Prerequisites

Having a valid machineId, corresponding long password and client_id
Permissioned for RDP News service
Python 3 installed
Jupiter Notebook or Jupyter Lab installed (or one can run the same code via plain Python)
Python libraries installed

Simple And Effective

A simple but very powerful use case of RDP News metadata is for lookup a news code or a news topic.

For example, we can send a request to lookup code B:227

And examine the result

}

Another is to find the codes that are related to a broader code, a.k.a. "children". For example, we request the children of code B:227

And the result contains the children

Lastly, we can just request a list of current metadata available, in portions, such as

The result will culminate in link next, for example:

That will allow us to retrieve more of the results, iteratively.

A More Involved Use Case

The combination of the last two simple use cases will allow us to implement a more involved use case, that we will go over next:

Retrieve all the metadata in use at the time,
Correlate the metadata in terms of relationship between the nodes in terms of multiple trees
Display the result as a simple non-graphical tree implementation AnyTree

We implement it in Jupiter Notebook, that can be downloaded from Example.RDPAPI.NewsMetadata on GitHub

A brief explanation on how the metadata is retrieved and correlated:

From RDP news metadata service, we retrieve the available metadata, and follow Next link recursively, till any Next data remains to be retrieved. These will become Nodes in the trees, the trees will define the relationships between the news codes that are Nodes.
For each news code/Node that has Children, we recursively retrieve Children until any Children remain, and when a new level is applied we take a note of the Parent of each news code/Node.
Once we have completed steps 1 and 2 full, we will have all the makings of our set of trees; and all that will remain is to structure them into a visual representation.

Let us step through the implementation.

1. Read Credentials

Prior to this step we must have the valid credentials assigned, password set and client_id established. Client_it is an API Key set up with API Key Generator with API selection "EDP".

For the quickest and surest way to get started, refer to Quickstart Guide for RDP RDP Quickstart Guide

2. Define Token Endpoint

3. Define Token processing functions

We are going to reuse token processing code that is also in use by RDP Quickstart and Python examples deck .

4. Obtain Valid Token

That wil be saved to a file for use by the consequent steps.

5. Request News Metadata

This step retrieves the very first portion of the metadata available. Please note the error handling implemented in this section, as the valid access token can potentially expire, we wish to handle a token expiry error by re-requesting a token on the spot and continuing, while handle any other error as a failure.

Parses it as json and optionally we can examine the results to be sure they are what we expect.

6. Request Children and Re-Categorize With Parent Information

We define a function processWithChildren that we will call recusrsively, on every item that is retrieved via metadata endpoint, exhaustively.

Next, we call it once:

And continue recusrively, while any more remains and Next link in the response remains non-empty. The output grows and reflects the processing that is taking place:

The completion is signalled with

7. Request Next on News Metadata and (optionally) Save to File

We may wish to save the resulting content to a file so we can search through it with a text editor, for debugging purposes, by default, we are keeping it commented

The tree of metadata that fully describes any currently published news content tends to be relatively large. During the processing, access token is likely to expire. Whenever this happens, we refresh the token and pick up the processing where we have left off.

8. Process into Tree Form

Lastly, we process all the retrieved data that includes children and is by now categorized in two lists NodesWithParents and NodesWithoutParents into a tree form and render the tree

For the purposes of simplicity, we render as simple AnyTree (find the documentation in references).

However, once the full content is retrieved, and structured, it can be presented in any tree form, plain text, graphical or even interactive tree representation. Our result looks simple:

And once refreshed to the latest, can be copied, pasted and conveniently text-searched

Conclusion

RDP News metadata service complements other RDP News services, allowing for more efficient consumption and analysis of headlines, stories and alerts, yielding itself easily to design and implementation of diverse use cases.