Contents

The article will be broken down into logical sections allowing easy access and reference when building out your search algorithms.

• Prerequisites
• Overview
• Search
  • How to approach Search
  • Query
  • Filter
  • Navigators
  • Sorting and Grouping
  • Properties / Meta data
    • Metadata
    • Debugging
    • Special Properties
    • Common Properties
  • Limits
• Examples

Prerequisites

The source code demonstrating the various examples are presented in both Python and C#/.Net within a Jupyter notebook. Refer to the GitHub Python or GitHub .Net projects outlining the prerequisites to run the examples.

Overview

The Search family of APIs exposes the desktop Search services through Refinitiv's Data Platform (RDP) so that it can be called by clients running outside the corporate firewall.

Search covers over:

• 77,000,000 quotes

• 35,000,000 instruments

• 9,000,000 organizations

• 2,000,000 people

• 1,000,000 deals

• 400,000 physical assets

Supporting the following key features:

• Freeform (Google-like) quick search

• Extensive filtering and ranking control

• Faceted navigation (supplemental information)

• Views to limit searches to a subset of content

• Metadata services supporting thousands of properties

All examples presented in this article use the Refinitiv Data Library for Python only.

Note Example packages will include both Python and .Net.

In this article and within the examples packages, I will be utilizing features and capabilities within the following both the Access and Content layers of the library.

Search

Searching for content can begin with the most basic, free form queries, to more complex filter expressions that can cross-reference the entire search data universe returning results matching your criteria. Content is organized within Views. A View defines properties, or fields, that can be grouped and sorted to suit your requirements.

Views

The data universe has been broken down into logical Views representing the entire data set. For example, if you wish to query for only people, a People View can be specified and a query can be applied only against that specific content set. Here is a list of available Views offered for users on the desktop:

Views offered to our Wealth clients (Light service):

Basic Views offered to all clients (also referred to as Search Explore):

The API also offers some help documentation that will provide a list of available Views. For example:

The details and examples within this article will demonstrate a number of different Views.

Note: The examples presented in this article use the full services within Search which are only available to the desktop user.  By default, users that have been granted any license to any RDP service will automatically be granted the Search Explore Views.  Our Wealth users will be provided the Search Light capabilities.  Both versions of search are a scale-down offering compared to the services available on the desktop.  

How to approach Search

Depending on the nature of data you wish to pull, Search offers different paths and capabilities to retrieve, filter, and group the desired result set. The more detailed your search specification, the more narrow and accurate the result returned. The following guidelines can be applied.

Note: The ordering of the steps below is only a recommendation if you are unfamiliar with the service or just don't know where to start. As you become more proficient, you will immediately know which step will be most suitable. In addition, as you gain experience, you will realize that there is more than one way to arrive at your result.

1. If possible, capture expected results
   
   If you have access to and are more comfortable using either the Eikon desktop or Refinitiv Workspace, you may be able to search for items within the interface or use Eikon Excel to extract expected results.  This will allow you to compare values within properties, navigators, or debug output that can provide up to hundreds of data points.  Doing this will give you a jump start on the steps outlined below.

2. Start with the most basic query

   Using a query expression will quickly give you a sense of what is returned and whether the result set appears to be relevant. For example, when searching for Organizations or People, you can perform queries like: "Microsoft", "IBM CFO", "Steve Jobs", etc. The complexity of your query will significantly affect the result set. For example, if I choose a vague term like "IBM", this will return over 40,000 hits. Using an unambiguous query like "IBM CFO" significantly narrows the result set down to just a few. Because query accepts any free-form text, you can be as vague or as explicit as you need.

   However, you may have search requirements where Query does not provide the desired output. Alternatively, your query may be too vague such that despite returning results, you are unable to lessen the scope of data returned. Whatever the reason, you can enhance your search by utilizing the Filter mechanism.

3. Filter the result set using specific properties

   One of the central search parameters is the ability to refine or filter your results using logical, structured expressions. The universe of data is organized within Views, each of which contains many properties. When you want to use the Filter mechanism to search, you apply structured expressions against Document Properties to help filter your data. Filters are a great way to refine your result set using simple, yet powerful expressions.

   The following is a simple expression to filter documents within a specific country:

   RCSCountryLeaf eq 'United States'

   Refer to the Filter section for more details and the syntax around this capability.

4. Navigators

   Where a Query and Filter will determine what is included within your results, Navigators provide the ability to categorize and summarize the results. For example, if I use a Query to list all ships and their hull types, that are out on the water, I will see hundreds of results. If I simply want a list of the different types of hulls for these ships, I can use a navigator to categorize and summarize this into logical buckets.

   Navigators can be used against a View, used in conjunction with either a query, a filter, or both. Navigators can be simple or very powerful but in general, allow you to collect and organize data within the result set and group that data into logical buckets.

   The power of Navigators will allow you to use the values returned within the buckets to use in your filters. In addition, Navigators offer additional features such as grouping and simple calculations to organize your buckets.

   Refer to the Navigators section for more details.

5. Use Views to narrow down your search

   By default, when you perform either a Query or Filter, you are searching against the "SearchAll" View which is a combination of all views within the universe. When you apply a query against a specific View that is relevant to what you need, this can significantly narrow your result set. For example, if I choose to search for 'IBM' using the default View (SearchAll), it will return over 40,000 documents. However, if I'm only interested in a specific View, such as "Instruments", my query will drop down to ~9500 documents. The more you become familiar with the details of the Views and what they offer, the better the results.

6. Selecting your Properties

   A Property represents a column or field for each document hit within the result set. By default, Search will return a well-defined set of Properties, such as DocumentTitle, PermID, etc. For example:

   	DocumentTitle	BusinessEntity	PermID	RIC	PI
   0	International Business Machines Corp, Public C...	ORGANISATION	<NA>	<NA>	37036
   1	International Business Machines Corp, Ordinary...	QUOTExEQUITY	55839165994	IBM	1097326

   When choosing your own Properties, you provide a comma-separated list. For example:

   "UnderlyingQuoteRIC, RIC, RCSAssetCategoryLeaf, TickerSymbol, UnderlyingIssuerName, ExpiryDateString"

   There can be up to hundreds of properties available for you to select when performing a search. The MetaData service provides a list of these properties for a specific View. However, depending on your result set, many properties may not be applicable or available for use. To help understand what properties are available and what their values contain, you can use a simple debugging capability to explicitly select all Properties available using the "_debugall" special property. For more details, refer to the Properties - Metadata section.

The above recommendations provide a great start when approaching search. As you educate yourself with the capabilities and the data, you will begin to master specific techniques and the suggestions outlined above. In addition, once you start seeing the results, you can further manipulate the organization by boosting, sorting, and grouping the data to suit your needs.

Query

The most basic search that can be performed is the query. The query is a free-formed text expression, similar to the type of query you can perform within Google, matching against documents within the specified view. For example, retrieve documents referencing the term: 'IBM'

A query matches on composite fields, which are internal and not listed by metadata.

Note: Using the debugging feature, you can see a list of these composite fields as identified with the term 'composite' in the name, e.g.: 'CompositeSearchAll'.

Using the above query, there are a few assumptions here regarding the default behavior. When searching, there are a number of possible parameters that can be applied to help narrow down the result set. To better understand, the following search command is equivalent to the one above:

By default, Search will occur against all Views, i.e. the entire data universe, returning the first 10 documents using a preset list of fields, or Properties, representing the above columns of data. Query does provide some flexibility to help narrow down the result set.

Query supports the following syntax:

Refer to the Playground Reference documentation for more details and specific behavior when applying query expressions.

While providing a query expression generally results in the desired output, there may be specific instances, depending on the View and what you are looking for, where you want to further refine the output based on the properties returned. Refer to the Filter section for more details.

Filter

One of the most powerful parameters when defining a search is the ability to filter the results. While a Query is more of an unstructured approach to search for text within documents, a Filter provides a well-defined structured approach when I know which fields I want to restrict within the matching documents.

The following is a basic example of a filter expression:

A filter expression takes on the basic form:

Like a Query, you can combine expressions with boolean operators as above with the example where I want to filter the search based on documents that contain the token 'LSEG' (London Stock Exchange Group) within the ticker symbol and that for each of these matches, the asset category is an ordinary share. Filters can and are often used in conjunction with a query expression within a search.

Note: The Playground Reference documentation defines the valid operator expressions such as 'eq', 'ne', 'xeq'. Each operator will define specific behavior, so it is important you review the details to ensure you have a complete understanding. For example, in the above expression that used the 'xeq' operator, the documentation states that only properties that are flagged with the 'Exact' attribute can be used. Unlike the 'eq' operator, 'xeq' cannot be used with wild cards and is case-sensitive.

Below, I have defined a search to look for the term 'IBM' within any document, but only collect results where the category is related to people.

While it can be fairly straightforward when building filter expressions, the most difficult challenge is how to determine the available properties to use and the values they contain?

Because the search facility is very rich in terms of how to formulate your expressions and the different paths you can take, in many cases, there can be multiple ways to arrive at your desired outcome. Whatever path you take to formulate your expressions, knowledge of which properties to use will be at the heart of your decision.

Defined within the Properties - Metadata section, I include a number of different techniques to help, not only the beginner, but advanced users to derive the properties to refine search results. You can be quite clever and creative when defining these expressions. The Filter section, defined within the Playground Reference provides a comprehensive outline of the types of expressions using well-defined syntax.

Navigators

Navigators provide the ability to summarize the distribution of your results. They are particularly useful when you are interested in gathering the domain of values for a specific property. Navigators can be used against a specific View, used in conjunction with either a query, a filter, or both. Navigators can be simple or very powerful, but provide a very useful way to capture results in logical buckets.

To demonstrate why you may want to use a Navigator as a preferred way to retrieve data, I will use the following query to retrieve the first 100 tankers to get a sense of what the different type of hulls there are and which types are most often used:

Here, I can see a list of the different tankers and their Hull Types. The above displays only 1 Hull Type: "Single", and a bunch of vessels that do not have a HullType association, i.e. <NA>.  I could expand the display to view the HullTypes within this collection but I'm only looking at the first 10,000 hits, which is the limit that I'm allowed to see in a single query.  I could use a GroupBy feature which would help narrow down the list but I may also want to capture the distribution of how many of each type are there. . To avoid this effort, I can instead use a Navigator.

Using the above query, I will apply a Navigator by selecting the 'HullType' property. What this does is instruct search to bucket all unique Hull Types and summarize the total found for each. 

The basic syntax of a Navigator is:

To retrieve all the available Hull Types using a Navigator:

Will return many documents, ranked by the symbols that are more significant or liquid. So in the above case, you will see the primary RIC code, "VOD.L", ranked at the top because it is the most significant instrument in the result set.

However, If I want to boost the result set for the above filter so that all the instruments that are traded on US exchanges appear at the top, I can apply a boost expression such as:

Will force all documents where 'VOD' traded on any US exchange will be boosted to the top.

While boosting can organize specific documents to appear at the top, I may choose to organize my documents based on a value in an ascending or descending order. To do this, I need to specify a list of properties using the order_by parameter. For example:

Is an expression that I can apply against a People View. The ordering expression will sort by the last name. The secondary property 'FirstName' is used to break a tie.

Finally, you may perform a query that lists duplicate items, such as a country or exchange. You can choose to group like properties together to help organize your output. For example:

Will collect all identical exchange codes within the output and group them together.

Additional details around these parameters are available within the Playground Reference documentation. To see a few of these in action, refer to the Examples section.

Properties / Metadata

The document properties or Properties represent the fields that define the data elements you want to retrieve as well as the elements you choose to filter your results. In addition, Properties are at the heart of other Search parameters such as Navigation, Grouping, and Boosting. There are hundreds of Properties available for each View providing critical details needed to fulfill your searching requirements.

In order for you to be successful at search, you will need to understand how to access the available Properties, how to determine what values they contain, and what they mean.

Metadata

The metadata service provides the ability to list all available properties within a View. To properly build a metadata list for inspection, you can issue the following:

In the above call, I'm requesting the list of Properties available within the View: GOV_CORP_INSTRUMENTS. Once executed, I can export it to an Excel spreadsheet for convenience:

The output will list all available properties and some important attributes for each. For example:

Each property determines how it can be used in the different parameters defined within Search. For example, some properties are sortable, navigable, searchable, etc. Refer to the Playground Reference documentation for a comprehensive outline of the output and a definition of each attribute.

Note: While the names of the properties do provide some reference as to their meaning, I understand that this may be challenging to fully comprehend what these properties mean. We are planning on enhancing the attributes to include important details such as a description. Until enhancements are available, you can infer the meaning based on the names of the properties as well as some debugging techniques defined below to better understand their values.

Debugging (_debugall)

While MetaData will help list all properties available within a View, all properties will not be available for use. That is, depending on the type of search you perform, will determine which properties will be populated - not all will be. This is where users can use a technique to list all available properties that are populated, for the specific search requested.

For example:

By selecting the special property called: "_debugall", this will list all properties and some internal meta-properties that provide additional debugging details. The key takeaway from this technique is the output will carry up to hundreds of available properties that are populated for your use - whether to be used for selection, filtering, or navigation. This technique can be extremely useful to identify values in certain properties.

Note - This technique should only be used for debugging. This setting does put an excessive burden on the service and should not be used for regular document processing. While this technique can be useful to help understand the availability and values of properties, the amount of output can be quite extensive, which is why the call above only requested 1 document.

Using Navigators to help understand values

When reviewing the debugging output, a number of common properties can be used when filtering your search. For example, you may want to filter your results based on certain categories or data belonging to a specific sector. Using navigators will help us understand the values contained within specific properties.

For example, when reviewing debug output from the above query, you may notice a very useful property called: SearchAllCategoryv3. This property helps us understand all available categories of documents the query term 'IBM' was found. To list these categories, use the following:

Output will list all categories such as: 'Equities', 'Futures', 'Commodities', 'Companies/Issuers', 'People', etc. From here, I can use the 'SearchAllCategoryv3' property to help filter out unwanted traffic. That is, if we're only interested in documents related to people, I can now enhance the search to look like this:

Special Properties

The Search service includes a number of special properties that enhance the ability to find meaningful results.

Common Properties

As you become more familiar with Search, you will soon recognize there will be a number of very useful properties that will act as your "go-to's" when filtering or selecting fields. Here are a few to get you started:

In addition to the above, there are a number of popular properties based on the Reuters Classification Scheme (RCS) that provide some critical tools when searching. When you review the metadata for a View or look at the output from a '_debugall' dump, you will notice many different RCS-based properties.

As outlined in Using Navigators to help understand values, you can list the values of a navigable property, such as an RCS-based property which provides critical details you can use. For example, to help understand values within an RCS-based property, you can apply a Navigator such as:

Many of these RCS-based properties are grouped to provide a logical organization you can use to help understand the nature of the data and how to utilize each to help narrow down your search results. The following table outlines the basic groupings for the different RCS-based properties:

Below are a number of common RCS-based properties:

Limits

Search is intended as an interactive tool rather than a facility for bulk processing. To avoid abuse and excessive heavy usage, the service does have upper limits in terms of the data set returned. Heavy usage can be a result of users blasting requests in a very short period of time or if the overall service is experiencing excessive load. Please refer to the reference guide within the playground for more details.

While it can be frustrating dealing with these limits, there are some possible suggestions that can be used to help.

Rate Limits

When running into issues related to overall load and heavy usage, users will need to throttle their requests. Issues such as blasting the service with too many requests in a short period of time can be managed. However, you will need to be aware that the service as a whole may have an excessive burden that may also affect the frequency and timing of your requests.

Upper Limits

The underlying service enforces an upper limit in terms of the results requested. In some cases, this can be managed through techniques such as grouping or by using navigators. However, if your requirement is to retrieve an excessive amount of results within a single call, you will likely need to be more creative as to how you can segment your search requests to capture the domain of content.

For example, if I choose to request for a list of all exchanges in the United States, I may start with a basic search like this:

This will list all the exchange codes, but because each matched document is an individual instrument, you will receive duplicate exchange codes. As a result, the upper limit will be hit.

While you can try to intelligently segment the requests to retrieve the whole universe, it is anything but an easy task. Instead, the preferred approach is to use a Navigator. Navigators are specifically designed to capture and summarize properties within logical buckets. For example, the following search will use a navigator to collect all unique exchange codes in the United States:

In the above request,  I'm explicitly specifying a 'top' of 0 because the result set is different than the output from a navigator - in my case, I don't need to see any hits, only the buckets defined within my navigator.  

The output will look something like this:

The above search request will provide a complete list of exchange codes. In addition, you can also observe the distribution, i.e. instance count, of each.

Segmenting the search

When I started with the first search to retrieve the list of exchange codes within the United States, I discovered that the result set returned the entire universe of instruments. If the goal is to capture the entire instrument list, I cannot group and bucket the result set as I did above. The # of hits returned is over 4 million so I am forced to go through a tedious process of segmenting the requests.

One way to do this is to choose some kind of indicator that will allow you to group your individual requests to successfully segment the result set. However, you need to first ask yourself - do I need the entire data universe? You may only be interested in a specific asset category thus reducing the universe of results significantly.

Refer to the Limits example for an outline on how to segment your search to avoid hitting limits.

While it may be possible to pull out excessive amounts of data, you should ask yourself if you need to do this. In most cases, you may be able to reduce the result set when you set up your search instead of pulling in everything and then massage the results once you have them in hand. Search was designed specifically to allow users to filter out unwanted content prior to returning the results. If you think this way through your searching patterns, you will undoubtedly avoid situations where you need to create complicated algorithms to unnecessarily pull excessive amounts of data. Whether narrowing the request based on the interested categories, or data for a specific region, you will find that you can significantly simplify your logic and avoid issues with limits.

Examples

The following table outlines the organization of the examples available within the GitHub Python or GitHub .NET code packages. Each code package contains a number of Jupyter Notebooks demonstrating some of the details outlined within this article.

Final Thoughts

Integrating search within your applications can be extremely powerful given the tools and content available. Looking at the example packages, you will notice there is more than one way you can arrive at your results. The tips and tricks offered within this guide will help you launch into search very quickly, but there is so much more that will help unleash the power of this service.

While this article focuses on some key techniques and different ways you can approach search, you likely want to better understand the finer details as you become more experienced with this service. The Playground Search reference documentation provides a detailed breakdown of some of the finer details that are mentioned in this article.

Acknowledgments

Putting this material together was by no means a singular effort. I want to thank Ray Tomkins for his help in providing some critical details around debugging, highlighting common elements, and the different ways to approach search. I also want to recognize the help of Simone da Costa and Toby Amos. Both were instrumental in helping me understand the general requirements around how our search is used and also contributing to a number of examples of typical workflows performed by the financial community.

Further Readings

Discover our new upcoming Refinitiv Data Platform Library - Part 1 of my RDP Library Introduction article

Q&A Forum for the Refinitiv Data Platform Library