Use Case Two
Refinitiv Data Platform News - Middle Office Compliance Use Cases
Introduction to Use Case
How a user can make a RDP API Request/Response call for news:
- Published in the last 24 hours, where
- The messages are tagged with PermIds (1, 2, 3, 4)
- And the messages contain topic codes (x, y or z)
To impelement the use case we are going to explore RDP news filters. RDP news filtering is the primary focus of this use case, while specific filter example is chosen to illustrate it.
Approach
We are going to use RDP API ( HTTP REST) calls directly, as opposite to using Refinitiv Data Library Endpoint representation or Eikon Data Library Python get_news method.
- Setup for RDP API Access
- Obtain a Valid RDP Authentication Token
- Request News Headlines - Define a Function
- Request Headlines per Filter Requirements (by PermIds and Topics, for the last 24 hrs
Request News Headlines - Define a Function
def getNewsHeadlines(query, numLines, return_as_text=False):
news_category_URL = "/data/news"
headlines_endpoint_URL = "/headlines?"
REQUEST_URL = base_URL + news_category_URL + RDP_version + headlines_endpoint_URL+ query # +"&limit="+ str(numLines);
accessToken = getToken();
print("Requesting: ",REQUEST_URL)
acceptValue = "*/*"
dResp = requests.get(REQUEST_URL, headers = {"Authorization": "Bearer " + accessToken, "Accept": acceptValue});
if dResp.status_code != 200:
print("Unable to get data. Code %s, Message: %s" % (dResp.status_code, dResp.text));
if dResp.status_code != 401: # error other then token expired
return("Error "+str(dResp.status_code))
accessToken = getToken(); # token refresh on token expired
dResp = requests.get(REQUEST_URL, headers = {"Authorization": "Bearer " + accessToken, "Accept": acceptValue});
if dResp.status_code == 200:
print("Resource access successful")
else:
print("Resource access successful")
if return_as_text:
return dResp.text
jResp = json.loads(dResp.text);
return jResp
next, we are going to use the defined function to
Request Headlines per Filter Requirements (by PermIds and Topics, for the last 24 hrs)
Some of the aspects to note:
- Max headlines request size at this time is defined at 100 headlines, that is also what is returned by default
- If the expected required result set of headlines is larger then 100, the result is paginated and the received portions of the result are assemebled into a single result dataframe
- Each headline contains a storyID which allows to retrieve the corresponding story
N = 100
query = 'query=(4295905573, 5030853586) and (searchIn:HeadlineOnly) and (last 24 hours)';
#query = 'query=(4295905573, 5030853586) and (Topic:E:F) and (searchIn:HeadlineOnly) and (last 24 hours)';
#query = 'query=(4295905573, 5030853586) and (Topic:SIG OR Topic:WEA) and (searchIn:FullStory) and (last 24 hours)';
#query = 'query=(4295905573, 5030853586) and (Topic:E:F) and (searchIn:HeadlineOnly) and (last 24 hours)';
jRespHeadlines = getNewsHeadlines(query, N)
dataAll = []
dfAll = pd.DataFrame()
iterations_count = 0;
while True:
cursor = ""
dfH = pd.json_normalize(jRespHeadlines,record_path =['data'])
if not dfH.empty:
dataAll.append(dfH)
if 'meta' in jRespHeadlines:
if 'next' in jRespHeadlines['meta']:
# print('Meta.next=', jRespHeadlines['meta']['next'])
cursor = urllib.parse.quote(jRespHeadlines['meta']['next'])
print('Iteration= ',iterations_count,'Cursor= ', cursor)
query = 'cursor='+cursor;
else:
print('No results that qualify filter requirements')
if not cursor:
break
iterations_count += 1
jRespHeadlines = getNewsHeadlines(query, N)
if dataAll:
dfAll = pd.concat(dataAll,ignore_index=True)
print('Displaying full result:')
display(dfAll)
we may wish to mark the iterations, for a large result set, this allows us to knowhow we progress:
Once the headlines result is received in total, and there is no more results to be received, that qualify our filter requirement, we assess the received result in dataframe:
the columns of the result that are of most interest to us are likley newsItem.contentMeta.subject and storyID. As newsItem.contentMeta.subject is what the story is about should allow us to further zero in on the news stories that we require and storyID is used to request the stories that we select:
next we can just simply request a story or stories per storyID
Request a Story - Define a Function
def getStory(storyId, jsonOrHtml):
news_category_URL = "/data/news"
story_endpoint_URL = "/stories"
REQUEST_URL = base_URL + news_category_URL + RDP_version + story_endpoint_URL+ "/" + storyId
accessToken = getToken();
print("Requesting: ",REQUEST_URL)
acceptValue = "application/json"
if jsonOrHtml != True:
acceptValue = "text/html"
dResp = requests.get(REQUEST_URL, headers = {"Authorization": "Bearer " + accessToken, "Accept": acceptValue});
if dResp.status_code != 200:
print("Unable to get data. Code %s, Message: %s" % (dResp.status_code, dResp.text));
if dResp.status_code != 401: # error other then token expired
return("")
accessToken = getToken(); # token refresh on token expired
else:
print("Resource access successful")
return dResp.text
we can display the story in Json format:
for instance, we can see that the story we have selected, is not part of Top News ( for more on Top News and Top News Images see References). Or display the selected story of interest to us in HTML:
Request Story for Display
txt = getStory(dfAll.iloc[[0]]['storyId'].item(), False)
from IPython.core.display import display, HTML
display(HTML(txt))
or apply any required formatting on display. To gain more insight into stories and associated rich metadata that is made available via RDP News service, we may wish to use tools such as Json Viewer, see References for more information.
We conclude our present brief discussion of RDP News filtering middle office compliance use case, to discuss more, you can always find us on Refinitiv Developers Forums
References
RDP News Metadata with JSON Viewer