A Step-By-Step Workflow Guide for RDP Client File Store (CFS) API

What is Delivery Platform (RDP) APIs?

The Delivery Platform - (RDP) provide various LSEG data and content for developers via easy-to-use Web-based API.

RDP APIs give developers seamless and holistic access to all of the LSEG content such as Environmental Social and Governance (ESG), News, Research, etc, and commingled with their content, enriching, integrating, and distributing the data through a single interface, delivered wherever they need it. The RDP APIs delivery mechanisms are the following:

• Request - Response: RESTful web service (HTTP GET, POST, PUT or DELETE)
• Alert: delivery is a mechanism to receive asynchronous updates (alerts) to a subscription.
• Bulks: deliver substantial payloads, like the end-of-day pricing data for the whole venue.
• Streaming: deliver real-time delivery of messages.

This example project is focusing on the Request-Response: RESTful web service delivery method only.

For more detail regarding RDP, please see the following APIs resources:

• Quick Start page.
• Tutorials page.

What is CFS?

Client File Store (CFS) aka File Distribution is a capability of Refinitiv Data Platform (RDP) that provides authorization and enables access to content files stored in publisher-supplied repository. CFS defines content ownership that publisher are isolated. And subscribers can trust the source of content.

CFS is engineered as a self-service metadata tool intend for publishers and subscribers. CFS provides bucket and file-set to organize files to simplify the interaction with publishers or subscribers CFS doesn't store file directly. Actual files are store in publisher-supplied. AWS S3 only one type storage that supported by current CFS.

Bucket

CFS facilitates buckets for use by Publishers to organize file-sets and files. Buckets store metadata about the files stored in publisher-supplied repositories. Buckets align with subscriptions and can contain multiple file-sets and files.

Publishers are responsible for creating buckets with the CFS API. This is a one-time process. The resulting bucket is owned one or more Publishers and is assigned a unique name that cannot be assigned to another bucket. A Publisher can have multiple buckets if they provide more than one dataset.

Claims are used to control access to buckets, file-sets and files. CFS does not manage or create claims, CFS only enforces them. Claims must be created in AAA. Subscribers must have at least one of the claims on the bucket in order to access the bucket.

Attributes are used to allow Subscribers to filter and search for content. Attributes are one method that a Subscriber can use to find files and/or file-sets.

Packages

A package is an indivisible set of file-sets that are all delivered together. Packages can consist of multiple file-sets that a grouping of file-sets.

The publisher will define the metadata for each package of content available to subscribers in CFS. Publishers are responsible for creating Packages and assigning claims to them.

Publisher need to create package first and then publisher able to create file-sets into Packages.

Fileset

A file-set is an indivisible set of files that are all delivered together. They can consist of multiple files that make up one large file or a grouping of files that represent related content. A file-set can also contain a single file. The Publisher decides the appropriate organization of their file-sets.

Publishers are responsible for creating file-sets into Packages. A Publisher can have multiple file-sets in a bucket.

Once all files have been added to the file-set and are ready for download, the Publisher updates the file-set status to READY. This enables the Publisher to control the release of their files and sets expectations for when the files will be available to subscribers. File-sets with a status of READY cannot be updated or modified. Updates must be published as a new file-set.

To access a file-set, Subscribers must have access to the bucket in which the file-set resides and have all of the claims associate with the file-set.

Files

Subscribers can only access the files to which they are entitled. On AWS S3, Subscribers can access files using a signed URI that redirects to the file on AWS S3 for downloading.

Files are available for a defined period of time that is determined by the Publisher.

Please find more detail on the CFS API User Guide document.

Prerequisite

Before I am going further, there is some prerequisite, dependencies, and libraries that the project is needed.

Access to the RDP with the your desire CFS file permission

This project uses RDP access credentials with the CFS file permission. You need both RDP User-ID (email base) and Machine-ID (GE-A-XXXXX) user types.

• User-ID: Identifier for user allowing access to contracted content and APIs on the API Playground page (https://apidocs.refinitiv.com/Apps/ApiDocs) or Bulk UI via Workspace/Eikon Desktop. It normally is email address of user (example: sample@lseg.com).
• Machine ID: Identifier for machine allowing access and run this tool or any automate applications. It normally is a username with "GE-A-XXXXXXXX" format.

Please see more detail about how to setup your RDP User-ID and Machine-ID from the Getting Started with Refinitiv Data Platform article.

Please contact your LSEG representative to help you with the RDP account and the bucket permission.

Internet Access

This demonstration connects to RDP on AWS via a public internet.

Python and Jupyter Notebook.

This project uses Python and Jupyter environment.

The Python Anaconda or MiniConda distribution/package manager is recommended.

RDP APIs Application Workflow

Step 1: Authentication with RDP APIs

Let’s start with the authentication. The CFS APIs are considered protected resources and require that your application be authenticated before making a data call. This authentication and provisioning of access token is based on OAuth 2.0 specification. The first step of an application workflow is to get a token from RDP Auth Service, which will allow access to the protected resource, i.e. data REST API.

The API requires the following access credential information:

• Username: The username/machine-id.
• Password: Password associated with the username/machine-id.
• Client ID: This is also known as AppKey, and it is generated using an App key Generator. This unique identifier is defined for the user or application and is deemed confidential (not shared between users). The client_id parameter can be passed in the request body or as an “Authorization” request header that is encoded as base64.
• Grant Type password: This is for initial authentication request. An application does not have any token, so it requests new tokens using username/password combination.

The HTTP request for the RDP APIs Authentication service is as follows:

Once the authentication success, the function gets the RDP Auth service response message and keeps the following RDP token information in the variables.

• access_token: The token used to invoke REST data API calls as described above. The application must keep this credential for further RDP APIs requests.
• refresh_token: Refresh token to be used for obtaining an updated access token before expiration. The application must keep this credential for access token renewal.
• expires_in: Access token validity time in seconds.

Next, after the application received the Access Token (and authorization token) from RDP Auth Service, all subsequent REST API calls will use this token to get the data. Please find more detail regarding RDP APIs workflow in the following resources:

• RDP APIs: Introduction to the Request-Response API page.
• RDP APIs: Authorization - All about tokens page.

Example source code in Python:

The next step is checking the authentication request result, then get the token information and assign them to local variables for the later use.

Now we have the access token information that can be used for further data requests.

Requesting Data from RDP APIs

That brings us to requesting the RDP APIs data. All subsequent REST API calls must include the Access Token via the Authorization HTTP request message header as shown below to get the data.

• Header:
  • Authorization = Bearer <RDP Access Token>

Please notice the space between the Bearer and RDP Access Token values.

If the token is still valid and has appropriate scope, the request is allowed to access the data.

The application then creates a request message in a JSON message format or URL query parameter based on the interested service and sends it as an HTTP request message to the Service Endpoint. Developers can get RDP APIs the Service Endpoint, HTTP operations, and parameters from Refinitiv Data Platform's API Playground page - which is an interactive documentation site developers can access once they have a valid Refinitiv Data Platform account.

Step 2: Listing the packageIds using the Bucket Name

Now let me move on to the CFS APIs call. To request the CFS data, the first step is to send an HTTP GET request to the RDP /file-store/v1/packages?bucketName={bucket-name} endpoint to list all Package Ids under the input bucket-name.

The HTTP Request structure is as follows:

The example bucket names for RDP content set are as follows:

Note: The bucket name is case-insensitive.

If you cannot find the bucket name for your interested content set, please contact your LSEG representative. I am using the bulk-ESG as an example bucket name for the Python code.

You can check the example data response to see the example of PackageIds from the following code:

Result:

Step 3: Listing the FileSets using the Bucket Name and Package ID

Now we come to getting the FileSets information. The application needs to send an HTTP GET request to the RDP /file-store/v1/file-sets?bucket={bucket-name}&packageId={packageId} endpoint to list all FileSets under the input bucket-name and packageId.

The HTTP Request structure is as follows:

This article uses 4037-e79c-96b73648-a42a-6b65ef8ccbd1 packageId of the bulk-ESG bucket as an example.

If you cannot find the package Id for your interested content set, please contact your LSEG representative.

Python code:

The FileSets response message from the API is as follows:

The File ID is in the files array above. I am demonstrating with the 4c35-1775-c1a590ea-8376-ac6c1546b908 file id.

Step 3.1: Listing the packageId using the Bucket Name - Paging

My next point is the paging feature. By default, the /file-store/v1/file-sets?bucket={bucket-name} endpoint always returns 25 results per request. You can adjust the number of return results via the pageSize query parameter, the maximum number is 100.

Let's try with pageSize=2 query as an example.

Python Code:

Now we get 2 entries per request from the API as we set via pageSize=2 parameter:

There is a @nextLink node contains the URL for requesting the next page of results with the following HTTP request.

Example of Python code:

Result:

Then you can continue to send requests to URL in @nextLink node to get the next page results.

Note: The /file-store/v1/packages?bucketName={bucket-name} endpoint on the step 2 above also supports the Paging feature with the same pageSize query parameter and @nextLink node.

Let’s leave the paging feature there.

Step 4: Get the file URL on AWS S3

Now let me turn to getting the file URL on Amazon AWS S3 service with the RDP /file-store/v1/files/{file ID}/stream endpoint.

The HTTP Request structure is as follows:

Python code:

The File URL is in the url attribute of the response message.

Result:

Step 5: Downloading the file

That brings us to download the file. Based on the S3 file_url above, the actual file name is Bulk-ESG-Global-Symbology-Organization-v1-Init-2023-11-26T16_04_11.525Z.jsonl.gz. So you need to replace the escape character %3A with _ (underscore) character.

Note: If you cannot download the file, please wait for a while and then retry download the file from the URL (file_url). Please do not flush the download requests. I am demonstrating with the polling2 library.

Python code:

Result:

Now you get the CFS file that you can extract and read the file.

That covers how to download the CFS file via the RDP APIs.

Step 6: Refresh Token with RDP APIs

Before the session expires (based on the expires_in parameter, in seconds) , an application needs to send a Refresh Grant request message to RDP Authentication service to get a new access token before further request data from the platform.

The API requires the following access credential information:

• Refresh Token: The current Refresh Token value from the previous RDP Authentication call
• Client ID: This is also known as AppKey, and it is generated using an App key Generator. This unique identifier is defined for the user or application and is deemed confidential (not shared between users). The client_id parameter can be passed in the request body or as an “Authorization” request header that is encoded as base64.
• Grant Type refresh_token: This is for getting a new Access Token.

The HTTP request for the RDP APIs Authentication service is as follows:

Once the authentication success, the function gets access_token, refresh_token, and expires_in from the RDP Auth service response message the same as the previous RDP Authentication call. An application must keep those value for the next Refresh Token call.

Caution: API Limit

The RDP Authentication service has the API limit described on the RDP APIs: Limitations and Guidelines for the RDP Authentication Service article. If the application flushes the authentication request messages (both password and refresh_token grant_type) beyond the limit, the account will be blocked by the API Gateway.

Python code:

Step 7: Revoke Token to ending the session.

This revocation mechanism allows an application to invalidate its tokens if the end-user logs out, changes identity, or exits the respective application. Notifying the authorization server that the token is no longer needed allows the authorization server to clean up data associated with that token (e.g., session data) and the underlying authorization grant.

The API requires the following HTTP Header and Credential parameter information:

• Header:
  • Authorization = Basic <App Key in Base64 format>

Please notice the space between the Basic and App Key in Base64 format values.

• Body parameter
  • token: The current Access Token value from the previous RDP Authentication call

The HTTP request for the RDP APIs Authentication service is as follows:

Python Code:

That’s all I have to say about the CFS API workflow.

How to run the demo application

The first step is to unzip or download the example project folder into a directory of your choice, then set up Python or Postman environments based on your preference.

Run the demo application

1. Open Anaconda Prompt and go to the project's folder.

2. Run the following command in the Anaconda Prompt application to create a Conda environment named CFS for the project.

3. Once the environment is created, activate a Conda CFS environment with this command in Anaconda Prompt.

4. Run the following command to the dependencies in the CFS environment

5. Once the dependencies installation process is success, run the following command to start the Jupyter Lab application.

6. Open a RDP-Generic-CFS.ipynb file and run each cell to learn the RDP CFS File workflow step by step.

That completes how to run an example Jupyter Lab notebook application.

Run the Postman LSEG CFS API Workspace

Now let me turn to the Postman collection. The RDP CFS API Postman Workspace is a public workspace in Postman that demonstrates how to use CFS API. It contains the following collection and environment:

• RDP CFS API Collection: This collection contain all HTTP requests with the RDP CFS file API. The requests are in ordered from Step 1 to Step 5.
• RDP CFS API Environment: This environment contains all parameters consumers need for requesting data with the requests in RDP CFS API Collection (including the credential).

Users can fork these Postman items to their own workspaces. Please follow these five steps to fork the RDP CFS Postman Workspace to a Postman workspace.

1. Create a new Postman workspace

Firstly, open the Postman application and then sign in to the Postman and create a new workspace.

Choose a Blank workspace, and click the Next button.

Set a workspace name, such as “My CFS Workspace”, set the Visibility to Personal, and then click the Create button.

The new workspace will appear on the list.

2. Open the LSEG CFS API Workspace

The LSEG CFS API Workspace is in the public workspace so you can search it in Postman by clicking on “Search Postman”.

And then, enter “LSEG CFS API”.

Click on the LSEG CFS API to open the workspace.

This LSEG CFS API workspace will be added to the workspace list.

3. Fork the RDP CFS API Collection

Select the LSEG CFS API workspace from the workspace list, and then select “...” menu of the RDP CFS API collection and choose the "Create a fork" command to fork this collection.

Then, enter the fork label, select the workspace (My CFS Workspace) created in the first step, uncheck “Watch original collection”, and then click on the Fork Collection button.

The forked collection will be created in the selected workspace.

4. Fork the Environment

Select the LSEG CFS API workspace from the workspace list, and then select “...” menu of the RDP CFS Environment and choose the "Create a fork" command to fork this environment.

Then, enter the fork label, select the workspace (My CFS Workspace) created in the first step, and then click on the Fork Environment button.

The forked environment will be created in the selected workspace. Input the USERNAME, PASSWORD, APP_KEY, and BUCKET_NAME variables with your RDP credential detail.

5. Running Postman Collection

The first step is run the "Step 1" Get Access Token using Password Grant (Machine ID)" request, please make sure that the environment on the top right corner is pointed to the RDP CFS Environment.

You can generate the source code from this Postman request by clicking the </> button and choose prefer language.

Once the Once authentication is successful, run the requests from step 2 to step 4

You can generate the source code from these Postman requests by clicking the </> button and choose prefer language on each request too.

If the Access Token is expired, the RDP CFS API requests return "token expired" error message as follows:

You need to refresh the Access Token by requesting "Step 5: Refresh Access Token using Refresh Token", then you can re-run the requests in step 2-4 again.

That covers how to run an example Postman collection.

Next Steps

That brings me to the end of this CFS API workflow project. The CFS API is a powerful API that allows developers to download the content file dynamically via a straightforward sequence of HTTP RESTful API calls. The demonstration code is written in Python but the concept of the API calls steps are the same for other programming languages.

At the same time, the Refinitiv Data Platform (RDP) APIs provide various LSEG data and content for developers via an easy-to-use Web-based API. The APIs are easy to integrate into any application and platform that supports the HTTP protocol and JSON message format.

You may interested in the following resources for more detail about the CFS data usage:

• How to use File Notification Message Distribution with the Client File Store (CFS) Service article.
• Find environmental footprint of your bond portfolio article
• RDP APIs Green Revenues CFS file Workflow - a dedicate Green Revenue CFS workflow
• RDP APIs ESG CFS file Workflow - a dedicate ESG CFS workflow

And much more on the Developer Portal website.

References

For further details, please check out the following resources:

• Refinitiv Data Platform APIs page on the LSEG Developer Community website.
• Refinitiv Data Platform APIs Playground page.
• Refinitiv Data Platform APIs: Introduction to the Request-Response API.
• Refinitiv Data Platform APIs: Authorization - All about tokens.
• Limitations and Guidelines for the RDP Authentication Service article.
• Getting Started with Refinitiv Data Platform article.
• CFS API User Guide.

For any questions related to Refinitiv Data Platform APIs, please use the RDP APIs Forum on the Developers Community Q&A page.