| Last update | Nov 2023 |
| Environment | Windows |
| Language | C# |
| Compilers | Microsoft Visual Studio 2019 |
| Prerequisites | DSS login, internet access, having done the previous tutorials |
| Source code | Download .Net SDK Tutorials Code |
Tutorial purpose
This is the sixth tutorial in a series of .Net SDK tutorials. It is assumed that the reader has acquired the knowledge delivered in the previous tutorials before following this one.
In this tutorial we build on the previous one by adding file input and output, and some error handling:
- Populate a large instrument list array read from a CSV file, and log input file errors to a file.
- Write the received data and extraction notes (including maintenance notes) to files.
- Analyze the extraction notes.
This sample contains 2 extraction requests: one with the ExtractWithNotes endpoint, the other with the ExtractRaw endpoint, to illustrate the differences between the two.
The first 3 added capabilities do not illustrate additional DSS REST API functionality, but serve to put it in the context of productized code. The last one illustrates an API capability that can be useful, depending on the use case.
Table of contents
- Getting ready
- Important warning on data usage
- Understanding the code
- DssClient.cs
- Program.cs
- Member declarations
- Initial file and directory management
- Creating the instrument list by reading an input file
- Creating the field name array
- Variant 1, using ExtractWithNotes:
- Creating and running an On Demand extraction (for JSON data)
- Retrieving the extracted data and notes, writing them to file
- Variant 2, using ExtractRaw::
- Creating and running an On Demand extraction (for compressed CSV data)
- Retrieving the compressed data, writing it to file
- Retrieving the extraction notes for the compressed data, writing them to file
- Code run results
- Understanding the refactored version
- Conclusions
Getting ready
Opening the solution
The code installation was done in Tutorial 1.
Opening the solution is similar to what was done in the previous tutorials:
- Navigate to the \DSS REST API\Tutorial 6\Learning folder.
- Double click on the solution file rest_api_more_advanced_topics.sln to open it in Microsoft Visual Studio.
Referencing the DSS SDK
Before anything else, you must reference the DSS REST API .Net SDK in the Microsoft Visual Studio project.
Important: this must be done for every single tutorial, for both the learning and refactored versions.
This was explained in the tutorial 2; please refer to it for instructions.
Viewing the C# code
In Microsoft Visual Studio, in the Solution Explorer, double click on Program.cs and on DssClient.cs to display both file contents. Each file will be displayed in a separate tab.
Setting the user account
Before running the code, you must replace YourUserId with your DSS user name, and YourPassword with your DSS password, in these 2 lines of Program.cs:
private static string dssUserName = "YourUserId";
private static string dssUserPassword = "YourPassword";
Important reminder: this must be done for every single tutorial, for both the learning and refactored versions.
Failure to do so will result in an error at run time (see Tutorial 1 for more details).
Important warning on data usage
DSS data request quota
DSS accounts have a quota of instruments that can be requested during a defined time period. The DSS subscription fees are proportional to the size of the quota. If the quota is reached, excess fees will be due. That situation must be avoided !
Data usage can be displayed in the DSS web GUI:
Contact your local account manager or sales specialist if you have queries on your DSS account quota or usage.
Instrument list files delivered with the tutorials
This tutorial (and the next one) handle large instrument lists, read from a file.
Care must be exercised when running these tutorials, to avoid over consuming data.
For this reason, several files are delivered with the tutorials code:
- DSS_API_1500_input_file.csv
- DSS_API_60_input_file.csv
- DSS_API_10_input_file.csv
The first one contains 1500 valid instrument identifiers, the second one 60, the last one only contains 10.
The text and illustrations that follow use the larger file. To avoid consuming too much data, we recommend you use the smaller file for your own tests, which will deliver similar results.
Understanding the code
We shall only describe what is new versus the previous tutorials.
DssClient.cs
This is the same as the refactored version of Tutorial 5, except for the leading comment, and an additional method at the end to create an EoD extraction that delivers compressed data, which we describe further down.
No additional explanations are required as the rest of the code was described in the previous tutorial.
Program.cs
At the top of the code we see a using directive was added for file input / output:
using System.IO;
Member declarations
We add a whole set of declarations, to define the input and output files and their locations. As output we will create a data file, a notes file and a separate error log file. As input we have an instrument identifier list:
private static string outputDirectory = "C:\\DSS REST API\\";
private static string dataOutputFile = outputDirectory + "DSS_API_tutorial_6l_output.txt";
private static string notesOutputFile = outputDirectory + "DSS_API_tutorial_6l_notes.txt";
private static string errorOutputFile = outputDirectory + "DSS_API_tutorial_6l_errors.txt";
private static string gzipDataOutputFile = outputDirectory + "DSS_API_tutorial_6l_gzipOutput.csv.gz";
private static string gzipNotesOutputFile = outputDirectory + "DSS_API_tutorial_6l_gzipNotes.txt";
private static string inputDirectory = outputDirectory;
private static string instrumentIdentifiersInputFile = inputDirectory + "DSS_API_10_input_file.csv";
The default directories and file names above can be used as is if you installed the tutorials code as described in Tutorial 1, using the directory structure defined in the Quick Start. If you are using a different folder structure, change the declarations accordingly.
The input file is a CSV file. A sample is included with the code, but you can also make your own.
- File format: 1 instrument per line.
- Line format: <identifier type>, <code>, <optional comment>
Example lines, first two are for a Cusip (CSP) and a RIC:
CSP,31352JRV1
RIC,ALVG.DE
CHR,0#.FCHI,CAC40
CIN,G93882192,Vodaphone
COM,001179594,Carrefour
ISN,CH0012138530,CS
SED,B1YW440,3i Group
VAL,24476758,UBS
WPK,A11Q05,Elumeo
The sample input files contains lines with correct line format syntax. They also include lines that are comments, empty or badly structured, to test the error handling included in the code.
To avoid consuming too much data, we recommend you use small files for your own tests. For more information, refer to the previous section: Important warning on data usage.
Initial file and directory management
We start with some basic input checks.
First we verify if the input file exists. If it does not, we display an error and exit the program:
if (!File.Exists(instrumentIdentifiersInputFile))
{
DebugPrintAndWaitForEnter("ERROR accessing " + instrumentIdentifiersInputFile +
"\nCheck if file and directory exist.");
return; //Exit main program
}
Then we verify if the output directory exists. If it does not, we display an error and exit the program:
if (!Directory.Exists(outputDirectory))
{
DebugPrintAndWaitForEnter("ERROR accessing directory " + outputDirectory + "\nCheck if it exists.");
return; //Exit main program
}
Then we clear the output files of any content left over from a previous run, and create them if they do not exist. Here is the code for one of the files (they all use the same code):
StreamWriter sw = new StreamWriter(dataOutputFile, false);
sw.Close();
Console.WriteLine("Cleared data output file " + dataOutputFile + "\n");
The second parameter of StreamWriter defines if content is to be appended to the file. By setting a value of false we clear the contents if the file already exists.
Next, we open the error output file and initialise it with a first message:
//Initialise the error output file:
sw = new StreamWriter(errorOutputFile, true);
sw.WriteLine("List of errors found in input file: " + instrumentIdentifiersInputFile + "\n");
We also open the input file:
StreamReader sr = new StreamReader(instrumentIdentifiersInputFile);
Creating the instrument list by reading an input file
Instead of creating an array of defined length like in the previous tutorial, we shall create a list for the instrument identifiers.
The reason is that, at this stage, we do not know how many instrument identifiers we will have, because:
- We don't know how many instruments are in the file.
- We shall validate the file entries, and only keep the good ones.
It is not possible to define an array of undefined size, and it would be a waste to define a huge one and then resize it.
So we define a list of undefined size, which we shall fill from the file. Once all instrument identifiers have been added to the list, we convert it to an array, as that is what the On Demand extraction API call requires.
The first step is to create the empty instrument identifiers list:
List<InstrumentIdentifier> instrumentIdentifiersList = new List<InstrumentIdentifier> ();
Then we populate the list, by reading one line at a time, from the input file. First we initilize some variables:
int fileLineNumber = 0;
string fileLine = string.Empty;
string identifierTypeString = string.Empty;
string identifierCodeString = string.Empty;
IdentifierType identifierType;
int i = 0;
Then we read all lines until we get to the end of the file, parse the data, and add the validated data to the list:
bool endOfFile = false;
while (!endOfFile)
{
//Read one line of the file, test if end of file:
fileLine = sr.ReadLine();
endOfFile = (fileLine == null);
if (!endOfFile)
{
fileLineNumber++;
try
{
//Parse the file line to extract the comma separated instrument type and code:
string[] splitLine = fileLine.Split(new char[] { ',' });
identifierTypeString = splitLine[0];
identifierCodeString = splitLine[1];
//Add only validated instrument identifiers into our list.
if (identifierTypeString != string.Empty)
{
if (identifierCodeString != string.Empty)
{
//DSS can handle many types, here we only handle a subset:
switch (identifierTypeString)
{
case "CHR": identifierType = IdentifierType.ChainRIC; break;
case "CIN": identifierType = IdentifierType.Cin; break;
case "COM": identifierType = IdentifierType.CommonCode; break;
case "CSP": identifierType = IdentifierType.Cusip; break;
case "ISN": identifierType = IdentifierType.Isin; break;
case "RIC": identifierType = IdentifierType.Ric; break;
case "SED": identifierType = IdentifierType.Sedol; break;
case "VAL": identifierType = IdentifierType.Valoren; break;
case "WPK": identifierType = IdentifierType.Wertpapier; break;
default:
identifierType = IdentifierType.NONE;
DebugPrintAndWriteToFile("ERROR line " + fileLineNumber +
": unknown identifier type: " + identifierTypeString, sw);
break;
}
if (identifierType != IdentifierType.NONE)
{
instrumentIdentifiersList.Add(new InstrumentIdentifier
{
IdentifierType = identifierType,
Identifier = identifierCodeString
});
Console.WriteLine("Line " + fileLineNumber + ": " + identifierTypeString +
" " + identifierCodeString + " loaded into array [" + i + "]");
i++;
}
}
Note: for more information on identifier types, refer to Tutorial 2: Creating an instrument list.
The code that follows is for error logging. Each error is displayed on screen and written to file, using a helper method:
//Error handling messages grouped here:
else
{
DebugPrintAndWriteToFile("ERROR line " + fileLineNumber +
": missing identifier code in line: " + fileLine, sw);
}
}
else
{
DebugPrintAndWriteToFile("ERROR line " + fileLineNumber +
": missing identifier type in line: " + fileLine, sw);
}
}
catch
{
DebugPrintAndWriteToFile("ERROR line " + fileLineNumber +
": bad line format: " + fileLine, sw);
}
}
else
{
if (fileLineNumber == 0)
{ DebugPrintAndWriteToFile("ERROR: empty file: " + instrumentIdentifiersInputFile, sw); }
}
} //End of while loop
The helper method which displays errors and writes them to file is defined at the end of the main code:
static void DebugPrintAndWriteToFile(string messageToPrintAndWriteToFile, StreamWriter sw)
{
Console.WriteLine(messageToPrintAndWriteToFile);
sw.WriteLine(messageToPrintAndWriteToFile);
}
Extract of the output for successfully added instrument identifiers (using the largest sample input file):
...
Line 1500: RIC ALVG.DE loaded into array [1497]
Line 1501: RIC IBM.N loaded into array [1498]
Line 1502: RIC 0001.HK loaded into array [1499]
...
Extract of the output for invalid lines in the largest input file:
...
ERROR line 1506: unknown identifier type: Junk
ERROR line 1507: missing identifier type in line: ,Junk
ERROR line 1508: missing identifier code in line: Junk,
ERROR line 1509: bad line format: Junk
ERROR line 1510: missing identifier type in line: ,
ERROR line 1511: missing identifier code in line: CSP,,
...
The error file contains all the ERROR messages that appear on screen, in the same format.
Back to the main code, the next step is to close the input file and the error output file:
sr.Close();
sw.Close();
This is followed with a check on the number of valid instrument identifiers:
int validIdentifiersCount = i;
if (validIdentifiersCount == 0)
{
DebugPrintAndWaitForEnter("Exit program due to no identifiers in the list.");
return; //Exit main program
}
Console.WriteLine("\n" + validIdentifiersCount +
" valid instruments were loaded into an array, outside of DSS,\n" +
"for use in the extraction.");
Result:
...
1500 valid instruments were loaded into an array, outside of DSS,
for use in the extraction.
...
Now we convert our instrument identifiers list to an array, as that is what the On Demand extraction API call requires:
//Convert the instrument identifiers list to an array:
InstrumentIdentifier[] instrumentIdentifiers = instrumentIdentifiersList.ToArray();
Creating the field name array
To create the field name array we proceed just like in the previous tutorials, calling the helper method we created in Tutorial 2:
string[] requestedFieldNames = CreateRequestedFieldNames();
We will use this array when we define the extraction.
No report template or schedule creation
Like in the previous tutorial, we do not create a:
- Report template, because the On Demand extraction calls implicitly define it.
- Schedule, because an On Demand extraction is on the fly.
Creating and running an On Demand extraction (for JSON data)
This is what we did in the previous tutorial. The extraction uses the ExtractWithNotes endpoint, which delivers the data in JSON format.
We call the helper method we created in the previous tutorial:
ExtractionResult extractionResult =
dssClient.CreateAndRunEodPricingExtraction(instrumentIdentifiers, requestedFieldNames);
DssCollection<ExtractionRow> extractionDataRows = extractionResult.Contents;
Retrieving the extracted data and notes, writing them to file
We call three helper methods:
DisplayAndLogExtractedDataFieldNames(dataOutputFile, extractionDataRows);
DisplayAndLogExtractedDataFieldValues(dataOutputFile, extractionDataRows);
DisplayAndLogAndAnalyzeExtractionNotes(notesOutputFile, errorOutputFile, extractionResult);
These methods are small variations of those we created in the previous tutorial to retrieve and display the data. They format the data as comma separated values (CSV).
The first difference with the previous tutorial lies in the fact that these will also write to file, so they have an additional input parameter (the data output file name), and 3 additional code lines, one to open the output file:
StreamWriter sw = new StreamWriter(outputFile, true);
One to write data to the file:
sw.WriteLine(returnedFieldNames.ToString());
And one to close the file afterwards:
sw.Close();
The second difference is that the extraction notes are parsed to detect if the extraction was successful, and if there were any warnings, errors or permission issues. This is based on simple string content analysis.
Here are 2 extracts of the results on screen, first the extracted field names:
...
Returned list of field names:
Instrument ID,Security Description,Universal Close Price Date,Universal Close Price,
Press Enter to continue
These are some of the (many) data values:
...
31352DCN8, FH S13 IO Fix IONtl, ,
31352DCR9, FH S16 IO Fix IONtl, ,
31352JRV1, FH S17 IO Fix IONtl, ,
ALVG.DE, ALLIANZ ORD, 6/8/2022 12:00:00 AM, 191.58
IBM.N, INTERNATIONAL BUSINESS MACHINES ORD, 6/8/2022 12:00:00 AM, 140.83
0001.HK, CKH HOLDINGS ORD, 6/8/2022 12:00:00 AM, 54.35
The data output file contains all the data lines, in exactly the same format, preceded with the list of field names.
Here we have 1500 instrument identifiers in the file, 1486 returned data. The missing 14 were not quoted on the day of the request, either because they are not volatile or maybe they are expired (like expired bonds or futures):
...
Extraction completed.
Number of data rows: 1500
Number of valid (non empty) data rows: 1486
...
The extraction notes are displayed and saved to file. Their analysis results are also displayed.
Creating and running an On Demand extraction (for compressed CSV data)
This is a different possibility we illustrate in this tutorial. Instead of retrieving JSON formatted data we want compressed CSV data, which is practical for storage. This also optimises extractions of large data sets.
Instead of the ExtractWithNotes endpoint (which delivers the data in JSON format), the extraction uses the ExtractRaw endpoint, which delivers compressed CSV formatted data.
Depending on your use case, you can choose either ExtractWithNotes or ExtractRaw.
We call a new helper method, that is very similar to the one we used previously:
RawExtractionResult rawExtractionResult =
dssClient.CreateAndRunEodPricingRawExtraction(instrumentIdentifiers, requestedFieldNames);
The difference between this helper method and the previous one is the endpoint, and the fact that it returns a raw extraction result. This also implies that we do not extract data rows from the extraction result contents.
Retrieving the compressed data, writing it to file
We must tell the server that we accept compressed data, by setting the Accept-Encoding header. If we do not do this, the server will send uncompressed data.
We then use a stream to retrieve the data and write it to a Gzipped file, which contains the data in CSV format:
extractionsContext.DefaultRequestHeaders.Add("Accept-Encoding", "gzip");
DssStreamResponse streamResponse = extractionsContext.GetReadStream(rawExtractionResult);
using (FileStream fileStream = File.Create(gzipDataOutputFile))
streamResponse.Stream.CopyTo(fileStream);
extractionsContext.DefaultRequestHeaders.Remove("Accept-Encoding");
Retrieving the extraction notes for the compressed data, writing them to file
The notes are available directly in the raw extraction result. We call a new helper method, which is practically the same as the one we used for the uncompressed data. The only difference is the 3rd input, which is a raw extraction result:
DisplayAndLogAndAnalyzeRawExtractionNotes(gzipNotesOutputFile, errorOutputFile, rawExtractionResult);
No cleaning up
As stated previously, cleanup on the DSS server is not required, as there is nothing to delete in this case.
Full code
The full code can be displayed by opening the appropriate solution file in Microsoft Visual Studio.
Summary
List of the main steps in the code:
- Authenticate by creating an extraction context.
- Check input file and output directory existence, clear output files.
- Create an array of financial instrument identifiers, populate it from a file. Manage and log errors.
- Create an array of field names.
- Create an extraction.
- Run the extraction, wait for it to complete. Retrieve the extracted data and extraction notes, display them and write them to files.
- Variant of step 6, for compressed data.
We do not create a report template or schedule, and there is no need to cleanup.
Code run results
Build and run
Don’t forget to reference the DSS SDK, and to set your user account in Program.cs !
Successful run
After running the program, and pressing the Enter key when prompted, the final result (using the largest input file) should look like this:
Returned session token: <token>
Press Enter to continue
Cleared data output file C:\DSS REST API\DSS_API_tutorial_6l_output.csv
Cleared notes output file C:\DSS REST API\DSS_API_tutorial_6l_notes.txt
Cleared error output file C:\DSS REST API\DSS_API_tutorial_6l_errors.txt
Cleared compressed data output file C:\DSS REST API\DSS_API_tutorial_6l_gzipOutput.csv.gz
Cleared compressed data notes output file C:\DSS REST API\DSS_API_tutorial_6l_gzipNotes.txt
ERROR line 1: unknown identifier type: Identifier Type
Line 2: CSP 911760HW9 loaded into array [0]
Line 3: CSP 911760JQ0 loaded into array [1]
Line 4: CSP 911760JR8 loaded into array [2]
Line 5: CSP 911760HP4 loaded into array [3]
Line 6: CSP 31359KUL9 loaded into array [4]
Line 7: CSP 31359KUJ4 loaded into array [5]
Line 8: CSP 31359KUK1 loaded into array [6]
Line 9: CSP 232928AE1 loaded into array [7]
Line 10: CSP 79548KHM3 loaded into array [8]
Line 11: CSP 911760JT4 loaded into array [9]
Line 12: CSP 911760KF2 loaded into array [10]
Line 13: CSP 911760KG0 loaded into array [11]
Line 14: CSP 92260MAP8 loaded into array [12]
Line 15: CSP 31359KGH4 loaded into array [13]
Line 16: CSP 31359KGJ0 loaded into array [14]
Line 17: CSP 31359KGK7 loaded into array [15]
Line 18: CSP 911760KT2 loaded into array [16]
Line 19: CSP 911760KU9 loaded into array [17]
Line 20: CSP 911760LB0 loaded into array [18]
...
This goes on with the next instruments of the file ...
The instrument identifiers import messages end like this, with quite a few errors generated by the wrongly formatted lines we included in the input file to test the error handling:
...
Line 1495: CSP 31352DCM0 loaded into array [1493]
Line 1496: CSP 31352DCN8 loaded into array [1494]
Line 1497: CSP 31352DCR9 loaded into array [1495]
Line 1498: CSP 31352JRV1 loaded into array [1496]
ERROR line 1499: bad line format: #RICs:
Line 1500: RIC ALVG.DE loaded into array [1497]
Line 1501: RIC IBM.N loaded into array [1498]
Line 1502: RIC 0001.HK loaded into array [1499]
ERROR line 1503: bad line format:
ERROR line 1504: bad line format: #Lines for error handling tests:
ERROR line 1505: unknown identifier type: Junk
ERROR line 1506: unknown identifier type: Junk
ERROR line 1507: missing identifier type in line: ,Junk
ERROR line 1508: missing identifier code in line: Junk,
ERROR line 1509: bad line format: Junk
ERROR line 1510: missing identifier type in line: ,
ERROR line 1511: missing identifier code in line: CSP,,
ERROR line 1512: missing identifier code in line: CSP,
ERROR line 1513: missing identifier type in line: ,CSP
ERROR line 1514: bad line format: CSP
ERROR line 1515: unknown identifier type: Junk
ERROR line 1516: unknown identifier type: Junk
ERROR line 1517: missing identifier code in line: RIC,,
ERROR line 1518: missing identifier code in line: RIC,
ERROR line 1519: missing identifier type in line: ,RIC
ERROR line 1520: bad line format: RIC
ERROR line 1521: unknown identifier type: Junk
ERROR line 1522: unknown identifier type: Junk
1500 valid instruments were loaded into an array, outside of DSS,
for use in the extraction.
We also created an array of field names, outside of DSS.
...
This is followed by the extraction, and the display of the returned field names and instrument data:
...
Next we will launch a direct on demand EOD pricing extraction,
using these 2 arrays.
Press Enter to continue
Please be patient and wait for the extraction to complete ...
Returned list of field names:
Instrument ID,Security Description,Universal Close Price Date,Universal Close Price,
Press Enter to continue
Returned field values:
======================
911760HW9, VNDE 963 1Z Z Fix, 6/8/2022 12:00:00 AM, 102.929810999
911760JQ0, VNDE 963 3 Vari, 6/8/2022 12:00:00 AM, 103.652327784
911760JR8, VNDE 963 4 Vari, ,
911760HP4, VNDE 963 IO Excess Vari IONtl, ,
31359KUL9, FN 96W3 X Excess Vari IONtl, ,
31359KUJ4, FN 96W3 A6 Seq Vari, ,
31359KUK1, FN 96W3 AL Seq Vari, ,
232928AE1, DRFC 94K1 A3 Fix, ,
79548KHM3, SBMSI7 936A B2 Sub Vari, 6/8/2022 12:00:00 AM, 99.227791996
...
This goes on with all the returned data, followed by the final statistics and messages:
...
31352DCK4, FH S10 IO Fix IONtl, ,
31282YAA7, FH S101 IO Fix IONtl, ,
31282YAB5, FH S102 IO Fix IONtl, ,
31282YAC3, FH S103 IO Fix IONtl, ,
31282YAD1, FH S104 IO Fix IONtl, ,
31282YAE9, FH S106 IO Fix IONtl, ,
31352DCL2, FH S11 IO Fix IONtl, ,
31352DCM0, FH S12 B Fix IONtl, ,
31352DCN8, FH S13 IO Fix IONtl, ,
31352DCR9, FH S16 IO Fix IONtl, ,
31352JRV1, FH S17 IO Fix IONtl, ,
ALVG.DE, ALLIANZ ORD, 6/8/2022 12:00:00 AM, 191.58
IBM.N, INTERNATIONAL BUSINESS MACHINES ORD, 6/8/2022 12:00:00 AM, 140.83
0001.HK, CKH HOLDINGS ORD, 6/8/2022 12:00:00 AM, 54.35
Extraction completed.
Number of data rows: 1500
Number of valid (non empty) data rows: 1486
Data was also written to file.
Press Enter to continue
The extraction notes and analysis are also displayed:
...
Extraction Notes:
=================
Extraction Services Version 16.0.43633 (806c08a4ae8f), Built May 9 2022 17:21:13
Holiday Rollover of Universal Close Price waived.
Processing started at 06/09/2022 09:33:47.
User ID: 9008895
Extraction ID: 595082235
Correlation ID: CiD/9008895/hlbllQ.080b9c88c4fdf683/RA/EXT.595082235
Schedule: _OnD_0x080b9c88c51df683 (ID = 0x080b9c88d04df683)
Input List (1486 items): _OnD_0x080b9c88c51df683 (ID = 080b9c88cdbdf683) Created: 06/09/2022 09:33:41 Last Modified: 0
6/09/2022 09:33:41
Schedule Time: 06/09/2022 09:33:41
Report Template (10 fields): _OnD_0x080b9c88c51df683 (ID = 0x080b9c88c52df683) Created: 06/09/2022 09:33:41 Last Modif
ied: 06/09/2022 09:33:41
Processing completed successfully at 06/09/2022 09:33:49, taking 2.08 Secs.
Extraction finished at 06/09/2022 08:33:49 UTC, with servers: x03A04, QSDHA1 (0.0 secs), QSHC12 (0.9 secs)
Usage Summary for User 9008895, Client 65508, Template Type EOD Pricing
Base Usage
Instrument Instrument Terms Price
Count Type Subtype Source Source
------- ----------------------------------- ---------------------------- -------------- ------------------------------
----------
1 Commercial Mortgage-Backed Security N/A N/A
1482 CMO's N/A N/A
3 Equities N/A N/A
-------
1486 Total instruments charged.
0 Instruments with no reported data.
=======
1486 Instruments in the input list.
No Evaluated Pricing Service complex usage to report -- 1486 Instruments in the input list had no reported data.
Writing RIC maintenance report.
Identifier,IdentType,Source,RIC,RecordDate,MaintType,OldValue,NewValue,Factor,FactorType
===============================================================================
SUCCESS: processing completed successfully.
===============================================================================
Press Enter to continue
We then run the second extraction, using the ExtractRaw endpoint, to retrieve compressed data. We save the file, then display and analyze the extraction notes:
...
We now run the same extraction, to retrieve Gzipped CSV data.
Please be patient and wait for the extraction to complete ...
Saved the compressed data file to disk:
C:\DSS REST API\DSS_API_tutorial_6l_gzipOutput.csv.gz
Press Enter to continue
Extraction Notes:
=================
Extraction Services Version 16.0.43633 (806c08a4ae8f), Built May 9 2022 17:21:13
Holiday Rollover of Universal Close Price waived.
Processing started at 06/09/2022 09:42:17.
User ID: 9008895
Extraction ID: 595083512
Correlation ID: CiD/9008895/hlbllQ.080bad00a66df6a6/RA/EXT.595083512
Schedule: _OnD_0x080bad00a67df6a6 (ID = 0x080bad00ab5df6a6)
Input List (1486 items): _OnD_0x080bad00a67df6a6 (ID = 080bad00aa8df6a6) Created: 06/09/2022 09:42:15 Last Modified: 06/09/2022 09:42:16
Schedule Time: 06/09/2022 09:42:16
Report Template (4 fields): _OnD_0x080bad00a67df6a6 (ID = 0x080bad00a68df6a6) Created: 06/09/2022 09:42:15 Last Modified: 06/09/2022 09:42:15
Processing completed successfully at 06/09/2022 09:42:18, taking 1.53 Secs.
Extraction finished at 06/09/2022 08:42:18 UTC, with servers: x03T01, QSDHA1 (0.0 secs), QSHC12 (0.4 secs)
Usage Summary for User 9008895, Client 65508, Template Type EOD Pricing
Base Usage
Instrument Instrument Terms Price
Count Type Subtype Source Source
------- ----------------------------------- ---------------------------- -------------- ----------------------------------------
1 Commercial Mortgage-Backed Security N/A N/A
1482 CMO's N/A N/A
3 Equities N/A N/A
-------
1486 Total instruments charged.
0 Instruments with no reported data.
=======
1486 Instruments in the input list.
No Evaluated Pricing Service complex usage to report -- 1486 Instruments in the input list had no reported data.
Writing RIC maintenance report.
Identifier,IdentType,Source,RIC,RecordDate,MaintType,OldValue,NewValue,Factor,FactorType
===============================================================================
SUCCESS: processing completed successfully.
===============================================================================
Press Enter to continue
We find the output files here; their contents reflect what was displayed on screen:
Potential errors and solutions
If the user name and password were not set properly, an error will be returned. See Tutorial 1 for details.
Understanding the refactored version
Explanations
DSS client helper class file: DssClient.cs
This has not changed.
Main program file: Program.cs
We enhance the error reporting. In the member declaration section, we add a list of possible errors, which we will use in input file error handling and reporting:
private enum Errors
{ NoError, EmptyFile, BadLineFormat, EmptyType, EmptyCode, UnknownIdentifier };
To check if the input file exists, we call a helper method:
if (!FileExists(instrumentIdentifiersInputFile)) { return; } //Exit main program
This helper method is declared in Program.cs, after the main code:
static bool FileExists(string fileName)
{
if (!File.Exists(fileName))
{
DebugPrintAndWaitForEnter("ERROR accessing " + fileName +
"\nCheck if file and directory exist.");
return false;
}
else { return true; }
}
To check if the output file exists, we call another helper method:
if (!DirectoryExists(outputDirectory)) { return; } //Exit main program
This helper method is declared in Program.cs, after the main code:
static bool DirectoryExists(string dirName)
{
if (!Directory.Exists(dirName))
{
DebugPrintAndWaitForEnter("ERROR accessing directory " + dirName + "\nCheck if it exists.");
return false;
}
else { return true; }
}
To clear the contents of the output files (if they exist), we call a helper method. Here is the code for one of the files (they all use the same code):
ClearFileContents(dataOutputFile);
Console.WriteLine("Cleared data output file " + dataOutputFile + "\n");
This helper method is declared in Program.cs, after the main code:
static void ClearFileContents(string fileName)
{
StreamWriter sw = new StreamWriter(fileName, false);
sw.Close();
}
To create the array of instrument identifiers, and populate it from the input file, we call a helper method:
InstrumentIdentifier[] instrumentIdentifiers =
PopulateInstrumentIdentifiersFromFile(instrumentIdentifiersInputFile, errorOutputFile);
This helper method is declared in Program.cs, after the main code. Note how error handling was restructured to use variable errorCode, to which we assign one of the values of the list we declared in the member declarations:
static InstrumentIdentifier[] PopulateInstrumentIdentifiersFromFile(
string instrumentIdentifiersInputFile, string errorOutputFile)
{
//Open the input file:
StreamReader sr = new StreamReader(instrumentIdentifiersInputFile);
//Initialise the error output file:
StreamWriter sw = new StreamWriter(errorOutputFile, true);
sw.WriteLine("List of errors found in input file: " + instrumentIdentifiersInputFile + "\n");
//Instead of an array (of defined length), create a list for the instrument identifiers.
//We do this because:
// we don't know how many instruments are in the file,
// we filter the file entries, to only keep the validated ones.
//Once the list is filled, we convert it to an array, as that is what the API needs.
//Create the empty instrument identifiers list:
List<InstrumentIdentifier> instrumentIdentifiersList = new List<InstrumentIdentifier> ();
//Populate the list, reading one line at a time from the file:
int fileLineNumber = 0;
string fileLine = string.Empty;
bool commaExistsInFileLine = false;
string identifierTypeString = string.Empty;
string identifierCodeString = string.Empty;
IdentifierType identifierType;
int i = 0;
//Loop through all lines until we get to the end of the file:
bool endOfFile = false;
while (!endOfFile)
{
Errors errorCode = Errors.NoError;
//Read one line of the file, test if end of file:
fileLine = sr.ReadLine();
endOfFile = (fileLine == null);
if (endOfFile && fileLineNumber == 0) { errorCode = Errors.EmptyFile; };
fileLineNumber++;
//Parse the file line to extract the comma separated instrument type and code:
if (errorCode == Errors.NoError && !endOfFile)
{
commaExistsInFileLine = (fileLine.IndexOf(",") >= 0);
if (commaExistsInFileLine)
{
string[] splitLine = fileLine.Split(new char[] { ',' });
identifierTypeString = splitLine[0];
identifierCodeString = splitLine[1];
}
else
{
errorCode = Errors.BadLineFormat; //Missing comma
identifierTypeString = string.Empty;
identifierCodeString = string.Empty;
}
}
if (identifierTypeString == string.Empty && errorCode == Errors.NoError)
{ errorCode = Errors.EmptyType; }
if (identifierCodeString == string.Empty && errorCode == Errors.NoError)
{ errorCode = Errors.EmptyCode; }
identifierType = IdentifierType.NONE;
if (errorCode == Errors.NoError && !endOfFile)
{
//DSS can handle many types, here we only handle a subset:
switch (identifierTypeString)
{
case "CHR": identifierType = IdentifierType.ChainRIC; break;
case "CIN": identifierType = IdentifierType.Cin; break;
case "COM": identifierType = IdentifierType.CommonCode; break;
case "CSP": identifierType = IdentifierType.Cusip; break;
case "ISN": identifierType = IdentifierType.Isin; break;
case "RIC": identifierType = IdentifierType.Ric; break;
case "SED": identifierType = IdentifierType.Sedol; break;
case "VAL": identifierType = IdentifierType.Valoren; break;
case "WPK": identifierType = IdentifierType.Wertpapier; break;
default: errorCode = Errors.UnknownIdentifier; break;
}
}
if (errorCode == Errors.NoError && !endOfFile)
{
//Add validated instrument identifier into our list:
instrumentIdentifiersList.Add(new InstrumentIdentifier
{
IdentifierType = identifierType,
Identifier = identifierCodeString
});
Console.WriteLine("Line " + fileLineNumber + ": " +
identifierTypeString + " " + identifierCodeString + " loaded into array [" + i + "]");
i++;
}
if (errorCode != Errors.NoError)
{
DebugPrintAndWriteToFileErrorMessage(
errorCode, fileLineNumber, fileLine, identifierTypeString, sw);
}
} //End of while loop
sr.Close();
sw.Close();
//Convert the instrument identifiers list to an array:
InstrumentIdentifier[] instrumentIdentifiers = instrumentIdentifiersList.ToArray();
return instrumentIdentifiers;
}
Note also that this helper method includes the list conversion to an array.
Finally, to save the compressed data file, we call a helper method:
dssClient.SaveCompressedData(rawExtractionResult, gzipDataOutputFile);
Here is the helper method:
public void SaveCompressedData(RawExtractionResult rawExtractionResult, string gzipDataOutputFile)
{
//We must tell the server we accept compressed data, otherwise it will send uncompressed data:
extractionsContext.DefaultRequestHeaders.Add("Accept-Encoding", "gzip");
DssStreamResponse streamResponse = extractionsContext.GetReadStream(rawExtractionResult);
using (FileStream fileStream = File.Create(gzipDataOutputFile))
streamResponse.Stream.CopyTo(fileStream);
extractionsContext.DefaultRequestHeaders.Remove("Accept-Encoding");
Console.WriteLine("Saved the compressed data file to disk:\n" + gzipDataOutputFile);
}
Full code
The full code can be displayed by opening the appropriate solution file in Microsoft Visual Studio.
Build and run
Don’t forget to reference the DSS SDK, and to set your user account in Program.cs !
Conclusions
This tutorial showed how instrument identifiers can be read from a CSV file, and how to log errors in the process. It also showed how extracted data could be written to a CSV file.
More importantly, it also compared the use of two endpoints; one that delivers JSON formatted data, for easy direct treatment, the other that delivers the data as a compressed CSV for effective download and storage of large data sets.
Now move on to the next tutorial, which continues building on the present tutorial by overcoming the limit on the number of instrument identifiers:
- Manage a list of more than 75000 instruments, using loops.