Article

Using .NET ODataCLI with DSS REST API ExtractWithNotes

November 29,2024

Author:

Jirapongse Phuriphanvichai

Open Data Protocol (OData) is an open protocol introduced by Microsoft in 2007. It defines a standard way for developing and consuming HTTP-based Web service APIs. This allows users to only focus on the business logic instead of worrying about other RESTful specifications. OData provides the service document that describes how a model constructs and the metadata document that contains a complete description of the feeds, types, properties, relationships exposed by the service. Both documents are defined by using an XML language.

Currently, Datascope Select and Tick History use OData to define HTTP-based REST APIs. The service and metadata documents are available on https://selectapi.datascope.refinitiv.com/restapi/v1.

This article explains how to use a .NET OData CLI tool to develop a C# application that extracts data from Datascope Select using the ExtractWithNotes endpoint. The application uses Visual Studio 2022 and .NET 8.0.

To install the .NET OData CLI tool and use it with the Datascope Select Authentication endpoint, please refer to the Using .NET ODataCLI with DSS REST API Authentication article.

C# Example: Datascope Select ExtractWithNotes

This section demonstrates how to use the OData CLI tool to generate classes from the OData metadata document of the Datascope Select Authentication and Extractions services. Then, use the generated classes to develop a C# application with Visual Studio 2022 (.NET 8.0) to extract data from Datascope Select.

Note: To install the .NET OData CLI tool and use it with the Datascope Select Authentication endpoint, please refer to the Using .NET ODataCLI with DSS REST API Authentication article.

1. Create a new C# project

First, create a new C# project on Visual Studio 2022 with .NET 8.0.

The project name is ODataDSSExtractWithNotes.

Select .NET 8.0.

Then, close the solution.

2. Generate proxy classes for the DSS authentication and Extractions services

Open a command line prompt and change the directory to the directory that contains the solution file (.sln). For this example, it is C:\examples\ ODataDSSExtractWithNotes.

Run the odata-cli tool to generate proxy client classes for the DSS Authentication service.

    	
            odata-cli generate -m "https://selectapi.datascope.refinitiv.com/restapi/v1/Authentication/$metadata" -o "ODataDSSExtractWithNotes" -fn "DSS_Authentication" -ns DSSAuthentication -s DSSAuthentication

Run the odata-cli tool to generate proxy client classes for the DSS Extractions service.

    	
            odata-cli generate -m "https://selectapi.datascope.refinitiv.com/restapi/v1/Extractions/$metadata" -o "ODataDSSExtractWithNotes" -fn "DSS_Extractions" -ns DSSExtractions -s DSSExtractions

The command uses the following parameters:

Parameter	Description
-m	The URI of the metadata document (The metadata of the DSS services)
-o	Full path to output directory (The project directory)
-fn	The name of the generated file (DSS_Authentication.cs and DSS_Extractions.cs)
-ns	The namespace of the client code generated (DSSAuthentication and DSSExtractions)
-s	Human-readable display name of the service instance (DSSAuthentication and DSSExtractions)

After running the commands, the required packages and resources are added to the project file (ODataDSSExtractWithNotes.csproj).

    	
            <Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.OData.Client" Version="8.2.1" />
    <PackageReference Include="Microsoft.OData.Core" Version="8.2.1" />
    <PackageReference Include="Microsoft.OData.Edm" Version="8.2.1" />
    <PackageReference Include="Microsoft.Spatial" Version="8.2.1" />
    <PackageReference Include="System.ComponentModel.Annotations" Version="5.0.0" />
    <PackageReference Include="System.Text.Json" Version="9.0.0" />
  </ItemGroup>
  <ItemGroup>
    <EmbeddedResource Include="DSSAuthenticationCsdl.xml" />
    <EmbeddedResource Include="DSSExtractionsCsdl.xml" />
  </ItemGroup>
</Project>

The DSS_Authentication.cs, DSS_Extractions.cs, DSSExtractions.Csdl.xml, and DSSAuthenticationCsdl.xml files are created in the project directory.

Open the project file. The DSS_Authentication.cs, DSS_Extractions.cs, DSSExtractions.Csdl.xml, and DSSAuthenticationCsdl.xml files are also added to the project.

The namespace of the generated proxy classes is DSSAuthentication and DSSExtractions respectively, as specified in the -ns parameter.

3. Use the generated DSS Authentication proxy classes

First, import the types defined in the DSSAuthentication namespaces.

    	
            using DSSAuthentication;

The DSS_Authentication.cs contains a service class (Authentication) derived from the DataServiceContext and the generated strong types defined in the DSS Authentication metadata document.

Initialize the Authentication class with the DSS authentication endpoint and then call the RequestToken method with the Credentials type to get a token.

    	
            try
{
    Authentication auth = new Authentication(new Uri("https://selectapi.datascope.refinitiv.com/restapi/v1/Authentication"));
    var resp = auth.RequestToken(new Credentials { Username = DSSUsername, Password = DSSPassword });
    var token = resp.GetValue();
 
    Console.WriteLine($"Token: {token}");
 
}
catch (DataServiceQueryException ex)
{
    Console.WriteLine(ex.InnerException?.Message);
    return;
}

Note: For more information regarding the Authentication service class, please refer to the Using .NET ODataCLI with DSS REST API Authentication article.

4. Use the generated DSS Extractions proxy classes

The example shows how to use the TermsAndConditionsExtractionRequest with the DSS ExtractWithNotes endpoint. It uses the JSON library (System.Text.Json) to serialize the generated strong types to JSON strings and then use the System.Net.Http.HTTPClient to send JSON strings to the DSS endpoints.

However, the types in the DSS_Extractions.cs file must be modified to ensure that the serialized JSON strings match the format accepted by the DSS.

Add the OdataType property and the JsonPropertyName attribute, as the first property, to the TermsAndConditionsExtractionRequest class

    	
                public partial class TermsAndConditionsExtractionRequest : ExtractionRequestBase
    {
 
        [JsonPropertyName("@odata.type")]
        public string OdataType { get; set; } = "#DataScope.Select.Api.Extractions.ExtractionRequests.TermsAndConditionsExtractionRequest";
…

Add the OdataType property and the JsonPropertyName attribute, as the first property, to the InstrumentIdentifierList class

    	
            public partial class InstrumentIdentifierList : InstrumentIdentifierListBase
{
…
    [JsonPropertyName("@odata.type")]
    public string OdataType { get; set; } = "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList";

Add the JsonDerivedType(typeof(InstrumentIdentifierList)) attribute to the SubjectIdentifierList class

    	
                [JsonDerivedType(typeof(InstrumentIdentifierList))]
    public abstract partial class SubjectIdentifierList
    {
    }

Add the JsonConverter(typeof(JsonStringEnumConverter)) attribute to all enumeration variables in the TermsAndConditionsCondition class. There are two enumeration variables in the TermsAndConditionsCondition class which are IssuerAssetClassType and FixedIncomeRatingSource

    	
            public partial class TermsAndConditionsCondition
    {
[JsonConverter(typeof(JsonStringEnumConverter))]
 public virtual global::DSSExtractions.DataScope.Select.Api.Extractions.ReportTemplates.IssuerAssetClassType IssuerAssetClassType
 {
}
 [JsonConverter(typeof(JsonStringEnumConverter))]
 public virtual global::DSSExtractions.DataScope.Select.Api.Extractions.ReportTemplates.FixedIncomeRatingSource FixedIncomeRatingSources
 {

Add the JsonConverter(typeof(JsonStringEnumConverter)) attribute to the IdentifierType enumeration variable in the SubjectIdentifier class

    	
            public partial class SubjectIdentifier
 {
…
        [JsonConverter(typeof(JsonStringEnumConverter))]
       
        public virtual global::DSSExtractions.DataScope.Select.Api.Content.IdentifierType IdentifierType
        {
…
}

In short, the TermsAndConditionsExtractionRequest class is modified by:

· Adding the OdataType property and the JsonPropertyName attribute to the TermsAndConditionsExtractionRequest and InstrumentIdentifierList classes

· Adding the JsonDerivedType attribute to the SubjectIdentifierList class to serialize it as the InstrumentIdentifierList class

· Adding the JsonConverter attribute to all enumeration variables used by the TermsAndConditionsExtractionRequest class to serialize enumeration values to strings.

Next, import the DSSExtractions.DataScope.Select.Api.Content, DSSExtractions.DataScope.Select.Api.Extractions.ExtractionRequests, and DSSExtractions.DataScope.Select.Api.Extractions.ReportTemplates namespaces.

    	
            using DSSExtractions.DataScope.Select.Api.Content;
using DSSExtractions.DataScope.Select.Api.Extractions.ExtractionRequests;
using DSSExtractions.DataScope.Select.Api.Extractions.ReportTemplates;

Then, initialize the TermsAndConditionsExtractionRequest class and set the IdentifierList, ContentFieldNames, and Condition properties. Next, use the JsonSerializer to serialize it to a JSON string.

    	
            TermsAndConditionsExtractionRequest termRequest = new TermsAndConditionsExtractionRequest();
 
InstrumentIdentifierList instruments = InstrumentIdentifierList.CreateInstrumentIdentifierList(true);
Collection<InstrumentIdentifier> list = new Collection<InstrumentIdentifier>();
list.Add(new InstrumentIdentifier { Identifier = "LSEG.L", IdentifierType = IdentifierType.Ric });
list.Add(new InstrumentIdentifier { Identifier = "GOOG.O", IdentifierType = IdentifierType.Ric });
list.Add(new InstrumentIdentifier { Identifier = "IBM.N", IdentifierType = IdentifierType.Ric });
instruments.InstrumentIdentifiers = list;
 
 
termRequest.IdentifierList = instruments;
termRequest.ContentFieldNames = new Collection<string> { "CUSIP", "SEDOL", "Exchange Code", "Currency Code" };
termRequest.Condition = new TermsAndConditionsCondition();
 
var request = new
{
    ExtractionRequest = termRequest
};
 
JsonSerializerOptions options = new JsonSerializerOptions();
options.WriteIndented = true;
options.DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull;
 
var requestStr = JsonSerializer.Serialize(request, options);

The JSON string will look like this:

    	
            {
  "ExtractionRequest": {
    "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TermsAndConditionsExtractionRequest",
    "ContentFieldNames": [
      "CUSIP",
      "SEDOL",
      "Exchange Code",
      "Currency Code"
    ],
    "IdentifierList": {
      "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
      "InstrumentIdentifiers": [
        {
          "Identifier": "LSEG.L",
          "IdentifierType": "Ric"
        },
        {
          "Identifier": "GOOG.O",
          "IdentifierType": "Ric"
        },
        {
          "Identifier": "IBM.N",
          "IdentifierType": "Ric"
        }
      ],
      "UseUserPreferencesForValidationOptions": true
    },
    "Condition": {
      "IssuerAssetClassType": "AllSupportedAssets",
      "ExcludeWarrants": false,
      "FixedIncomeRatingSources": "None",
      "UseRelativeAnalytics": false
    }
  }
}

Please make sure that the “@odata.type” properties are in the request message and all enumeration properties contain strings.

Then, create a HttpRequestMessage with the Post method to the ExtractWithNotes endpoint, set the Prefer and Authorization headers, and set the JSON string in the request’s body.

    	
            HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Post, "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractWithNotes");
 
requestMessage.Headers.Add("Prefer", "respond-async");
requestMessage.Headers.Add("Authorization", "Token " + token);
 
 
requestMessage.Content = new StringContent(jsonStr, Encoding.UTF8, "application/json");

Next, create a HttpClient to send the request message and then check the status of the HTTP response. If the status code is Accepted (202), it will create another HttpRequestMessage with the GET method to poll the extraction status. Otherwise, it calls the EnsureSuccessStatusCode method to check the status code of the HTTP response and throws an exception if the IsSuccessStatusCode property for the HTTP response is false.

    	
            HttpClient client = new HttpClient();
var response =  client.Send(requestMessage);
 
while(response.StatusCode == System.Net.HttpStatusCode.Accepted)
{
    Console.WriteLine(response.Headers.GetValues("Location").First());
    var requestLocation = new HttpRequestMessage(HttpMethod.Get, response.Headers.GetValues("Location").First());
    requestLocation.Headers.Add("Prefer", "respond-async");
    requestLocation.Headers.Add("Authorization", "Token " + token);
    Thread.Sleep(2000);
    response = client.Send(requestLocation);
}
 
try
{
    response.EnsureSuccessStatusCode();
}
catch (HttpRequestException ex)
{
    Console.WriteLine(ex.Message.ToString());
    Console.WriteLine(response.Content.ReadAsStringAsync().Result);
    return;
}

Finally, use the JsonSerializer to deserialize the response’s content to a JsonNode and retrieve data from the JsonNode.

    	
            var strResponse = response.Content.ReadAsStringAsync().Result;
 
JsonNode responseJSON = JsonSerializer.Deserialize<JsonNode>(strResponse);
 
JsonArray array = responseJSON["Contents"] as JsonArray;
 
Console.WriteLine(responseJSON["Notes"]);
 
foreach (JsonObject row in array)
{
    Console.WriteLine("");
    foreach(var value in row.AsEnumerable())
    {
        Console.WriteLine($"{value.Key}:{value.Value}");
    }        
}

The output is:

    	
            IdentifierType: Ric
Identifier: LSEG.L
CUSIP:
SEDOL: B0SWJX3
Exchange Code: LSE
Currency Code: GBp
IdentifierType: Ric
Identifier: GOOG.O
CUSIP: 02079K107
SEDOL: BYY88Y7
Exchange Code: NSQ
Currency Code: USD
IdentifierType: Ric
Identifier: IBM.N
CUSIP: 459200101
SEDOL: 2005973
Exchange Code: NYS
Currency Code: USD

The example is available on GitHub.

Summary

OData is an open standard for defining HTTP-based web service APIs. It defines a metadata document that provides a complete description of the service including its types, and functions. Odata-cli is a .NET tool that can parse a metadata document and generate a DataServiceContext class with all strongly typed data types defined in the web service.

OData is used by the Datascope Select and Tick History products to define HTTP-based REST APIs, allowing the odata-cli tool to generate the C# service classes and C# strongly typed data types for the DSS and Tick History REST APIs. To use the generated classes with the JSON and HttpClient libraries for data extraction, they must be modified to include the JsonPropertyName, JsonDerivedType, and JsonConverter attributes, ensuring that the serialized JSON strings match the format accepted by the DSS. Then, deserialize a returned HTTP response to the JsonNode, JsonArray, and JsonObject classes to retrieve data.

References

1. OData. (n.d.). OData: The Protocol for Building and Consuming RESTful APIs. Retrieved October 21, 2024, from https://www.odata.org

2. Wikipedia contributors. (n.d.). Open Data Protocol. In Wikipedia, The Free Encyclopedia. Retrieved October 21, 2024, from https://en.wikipedia.org/wiki/Open_Data_Protocol

3. Microsoft. (n.d.). Getting started with OData CLI. Retrieved October 21, 2024, from https://learn.microsoft.com/en-us/odata/odatacli/getting-started

4. LSEG. (n.d.). Using .NET ODataCLI with DSS REST API Authentication. London Stock Exchange Group. Retrieved November 20, 2024, from https://developers.lseg.com/en/article-catalog/article/using--net-odatacli-with-dss-rest-api-authentication