Article
Accelerating Search: Unlocking the Power of Property Exploration
As outlined in my Building Search into your Application Workflow article, LSEG provides a wealth of financial information through its Search service covering content such as quotes, instruments, organizations, people, and many more assets that users can programmatically access and integrate within their business applications. While Search provides a significant amount of content, power, and flexibility, there are challenges when attempting to navigate through the hundreds of available financial properties when deciding how to extract data. In that article, I provide guidance and tips to determine the possible approaches, from simple discovery through experimentation to more advanced techniques such as generating debug output and extracting metadata details. While the suggestions are helpful and effective, the work involved can be challenging and intimidating.
The following article outlines a value-add API, called SearchPropertyExplorer, that greatly simplifies the challenges of discovering financial properties when programmatically building Search. While the techniques outlined in the Search article demonstrate some extremely useful tips, this valid-add API will alleviate the need to extract the detailed plumbing necessary to discover relevant properties. The goal of this API is to elevate the ability to understand and construct the necessary queries to pull out desired content within the LSEG ecosystem. Think of this API as a powerful debugging mechanism to understand the data within Search.
Once a user determines the key attributes and properties available to them, they can then apply this discovery within the Search API to pull down desired data.
Contents
The article will focus on a specific Use Case that demonstrates the key capabilities of the service as well as other useful features.
Prerequisites
The source code demonstrating the SearchPropertyExplorer API is available on GitHub. The source code example demonstrates a number of features outlined within this article. You must have access the LSEG Workspace desktop application and the ability to run within a Python Jupyter environment or using the LSEG desktop CodeBook app available within LSEG Workspace.
Getting Started
The SearchPropertyExplorer API is a simple Python package, or module, included within the LSEG Data Library for Python, that accepts criteria allowing users to easily discover properties and values. As we walk through some examples, you will better understand how and where this tool will help in your construction of queries and the extraction of values that will greatly accelerate the goal of retrieving desired content. To get the most out of this article, I would recommend you have a basic familiarity with Search and basic details outlined within the Search Article referenced above.
# To get started, import the LSEG Data Library for Python and expose the Search Property Explorer interfaces.
import lseg.data as ld
from lseg.data.content import search
from lseg.data.discovery import SearchPropertyExplorer, PropertyType
ld.__version__
Create a Search Property Explorer
The SearchPropertyExplorer module is a rich interface that is responsible for managing the execution and exploration of interrogating search properties. The goal of this module is to hold financial properties, metadata, and data values, based on search criteria, that users can observe and interrogate at any time. To get started, simply create an instance of the SearchPropertyExplorer object. A single instance can be used throughout your interrogation - this is a one-time action.
# Create our explorer defined within the library. This acts as the entry point for exploration.
explorer = SearchPropertyExplorer()
Use Case
The simplest way to demonstrate what this module will provide is to define a legitimate use case that outlines the journey of what is involved in determining and discovering the appropriate search criteria and properties to solve a problem.
For example, we'll define the following use case:
- Retrieve a list of active bonds for a specific organization, eg: Santander Bank
- For this organization, only request bonds issued within a specific country, eg: United States
- For each bond, retrieve details such as:
- Maturity date
- Issue Date
- Coupon Rate
- Coupon Type
- Amount Outstanding
- Amount Issued
Important Note: I'm not interested in Perpetual Bonds - these do not have a maturity date.
Suggested approach
The best way users typically approach dealing with the above use case is to refer to an existing example and build from there. This will jump-start the journey to give a sense of what is returned. However, once you realize you need additional information or the data returned isn't exactly what you were looking for, you will be forced to interrogate the service. That is, you will typically go through an iteration of actions of guessing at criteria in hopes you find the answer.
As part of these actions, you will likely perform specific steps such as:
Generating criteria using Workspace Advanced Search
To help accelerate the journey, users can utilize the 'Export Query' option within Advanced Search defined within Workspace. In some cases, this may solve a number of desired queries, but to fine-tune criteria, the challenge still exists and likely will involve the following 2 steps.
Extract the metadata
This provides a complete list of properties and additional attributes about each field.
Retrieve debug output
Debug output is a dump of values associated with a specific query. As the Search Article outlines, the output will contain hundreds of properties and their values for you to interrogate.
In the latter 2 steps, you will likely need to export the data to an editor or application like Excel to perform exhaustive, manual interrogation. While these steps are useful, they are very cumbersome, especially when you need to go through repetitive tests using different views and search criteria.
To alleviate the burden, the SearchPropertyExplorer module provides a path that will greatly simplify the interrogation of properties and the values they contain. As you will discover below, this will rapidly accelerate the repetitive steps necessary when defining criteria.
Based on the above use case, the following steps are recommended:
Step 1 - Start with a basic query
The SearchPropertyExplorer interface supports the specification of a basic query expression as a quick way to retrieve related content. While this is a simple and convenient way to get started, you may already have a jump-start with either an existing example or by utilizing the Advanced Search query export described above. Either way, this step is used to define a quick way to get some relevant data from the service.
Step 2 - Filter out unwanted data
Once a data set has been generated, we begin to narrow down the result set by specifying filter expressions. This step is critical as it leads to step 3 below. In some cases, we may not discover the appropriate output properties unless we properly filter our request first. For example, in our use case, we must ensure our results do not include perpetual bonds. If they did, we may not be able to discover a maturity date property - perpetual bonds have no maturity.
Step 3 - Define the output properties
Once we have defined a relevant filter expression, we can begin the process of discovering the output properties we want to capture.
Step 1 - Start with a basic query
When using SearchPropertyExporer, the goal is to feed the valid-add API search criteria. Initially, you will likely begin with simple, google-like, query expressions. As you become more advanced, you will move to more complicated filters and even navigators.
# The SearchPropertyExplorer interface provides the same criteria as the Search service to request for data.
# Let's start with a basic query...
props = explorer.get_properties_for("Santander bonds")
props.df
The above execution will perform the steps to query the Search service, retrieve debug and metadata information and prepare the results within a dataframe. Effectively, a combination of the metadata and debug steps mentioned above. At this stage, you can quickly scan the output above to verify relevancy of the fields.
Aside from providing the property details, the get_properties_for() method returns a properties object that provides a number of important mechanisms to interrogate the result set. One key attribute is the hits count.
props.hits_count
63536
Hits count
This value represents the number of rows, or hits, found within the back-end system based on your search criteria. When search organizes the result set, by default, the service will rank the results automatically. When using the Search API, the 1st hit is the most relevant based on the query and filter expressions within the request. When preparing and presenting data within the SearchPropertyExplorer, the most relevant hit will always be returned. For example, the above result represents all properties defined within the highest ranked hit.
The goal at this point is to determine if the details within the table relate to your requirements. The hits count is a great way to continually measure whether the query satisfies requirements. Because we provided such a broad query to the service, the # of hits is massive - which is likely not desirable, especially if the requirements are specific. We can change the query expression to provide more relevant results or provide specific filters. The ultimate goal of exploration is to narrow down your search so the hits count represents a reasonable, expected result set.
Step 2 - Filter out unwanted data
In this step, the goal is to remove unwanted hits. For example, we are only interested in active bonds that have been issued in the United States. In addition, our requirement is to filter out perpetual bonds. As part of this exercise, we will continually update the search criteria and perform multiple executions until we believe our criteria matches our requirements.
The following interrogation steps will look for properties that will help us create filter expressions to narrow down our result set.
As outlined within the Search Article, the Search ecosystem defines logical views representing the entire data set available within the service. It is extremely important you generally understand how views can alter the type of data you retrieve. For example, the above execution to retrieve 'Santander bonds' will search across the entire Search content set - this is represented by the default logical view called: 'SearchAll'. Doing so, may not only return bond instruments but other data, such as bond pricing information. Given this, there is a logical view, GovCorpInstruments, that will narrow down the result set to provide only the bond, not the individual pricing that is included within the view GovCorpQuotes.
# First, narrow down the data set by limiting the search within a specific view
props = explorer.get_properties_for(
view = search.Views.GOV_CORP_INSTRUMENTS,
query = "Santander bonds"
)
props.hits_count
33314
The above criteria limits the result to provide instruments (bond) only. You can see how the results hit count has significantly dropped down from above. Let's continue...
# Filter out perpetual bonds. To do this, let's see if a relevant property exists...
matches = props.get_by_name("perpetual")
matches.df
The get_by_name() method provides a simple way to discover relevant fields available within the properties collection. We are presented with all properties matching our input expression: 'perpetual'. We can clearly see our match is a suitable candidate that indicates whether the bond is perpetual or not. We can use this value to filter out those bonds that are perpetual.
# Update the execution to filter out perpetual bonds...
props = explorer.get_properties_for(
view = search.Views.GOV_CORP_INSTRUMENTS,
query = "Santander bonds",
filter = "IsPerpetualSecurity ne true"
)
props.hits_count
33196
Continuing on, we can further filter those bonds that have not yet matured and are considered 'active'.
Let's interrogate the explorer to find these properties.
Note: As outlined within the Common Properties section of the Search article, two useful properties, IsActive and AssetState can be used to help ensure the bonds we request for are active and valid.
# Let's determine if we have an 'active' property...
matches = props.get_by_name("active")
matches.df
# As well as a 'state' property...
matches = props.get_by_name("state")
matches.df
The above expression to determine if there are any 'state' properties returned no matches.
As the Search Article outlines, in some cases, the 'active' or 'state' properties may not be available. Typically, the AssetState does provide more granularity about the state of the instrument, and that you can use this one as opposed to relying on the boolean state of IsActive. In our case, only one is available and thus we can simply use the following filter expression:
"IsActive eq True"
The above expression filters bonds that are active.
# Next, how do we figure out bonds that are issued within a specific country?
# Let's first see if there is a property that will help.
matches = props.get_by_name("issuer country")
matches.df
Based on our interrogation, we see a number of candidates that can help. For example, the RCS-based properties presented above will allow us to narrow down which country the bond was issued. However, if we choose the property RCSIssuerCountryLeaf, we need to make sure the spelling is accurate. For example, we are interested in bonds issued in the U.S. How do we figure out the filter expression? That is, do we look for 'USA'? 'United States'? or 'US'? One way may that may work is to update our criteria to search for 'Santander bonds USA'.
However, there is a better way - Navigators.
Navigators provide the ability to summarize the distribution of your results. I would suggest you refer to the Navigators writeup for more details and examples of them in use.
# The RCSIssuerCountryLeaf, or RCSIssuerCountry, properties are navigable.
# Let's perform an execution and show what we can do...
props = explorer.get_properties_for(
view = search.Views.GOV_CORP_INSTRUMENTS,
query = "Santander bonds",
filter = "IsPerpetualSecurity ne true and IsActive eq true",
navigators = "RCSIssuerCountryLeaf"
)
# Display the result of the above execution showing the results of the navigation...
props.navigators
Not only do we get a list of all countries within our results, but a distribution of hits for each. For our use case, we can update the filter expression to ensure we are listing bonds that are issued within the United States. At this point, it is worth noting within the previous table showing the different properties, the column titled Exact contains a value of True for the property 'RCSIssuerCountryLeaf' which means we can search for countries that exactly match our expression.
At this point, we should have a fairly accurate request that will pull down the list of relevant bonds.
# Add our country filter using an exact match expression...
props = explorer.get_properties_for(
view = search.Views.GOV_CORP_INSTRUMENTS,
query = "Santander bonds",
filter = "IsPerpetualSecurity ne true and IsActive eq true " \
"and RCSIssuerCountryLeaf xeq 'United States'"
)
props.hits_count
86
Step 3 - Define the properties (output) we want to retrieve
Now that we generally have the set of bonds of interest, let's go through the action of determining the output fields, based on our use case. The steps involved here to interrogate the service are the same as we outlined in Step 2 above.
Just as a reminder, we want to capture the following:
- Maturity date
- Issue Date
- Coupon Rate
- Coupon Type
- Amount Outstanding
- Amount Issued
# Locate Maturity Date...
matches = props.get_by_name("maturity date")
matches.df
# Issue date...
matches = props.get_by_name("issue date")
matches.df
# Coupon rate...
matches = props.get_by_name("coupon rate")
matches.df
# Coupon type...
matches = props.get_by_name("coupon type")
matches.df
# Amount outstanding...
matches = props.get_by_name("amount outstanding")
matches.df
# Amount issued...
matches = props.get_by_name("issued")
matches.df
Step 4 - Putting it altogether
While going through the above exercise, I was able to successfully locate the relevant required properties. However, this exercise did require some experimentation. The biggest challenge with Search is how to figure out the names of the properties. Given there are hundreds available, having the ability to quickly search for them will be invaluable. Because of this, you will likely play with the expressions you type. The nice thing is that you can at least see the values associated with the properties displayed in order for you to better understand the meaning behind the property.
Now that we've retrieved our output fields, let's create a search request. We'll also order the results based on the bond maturity date.
ld.discovery.search(
view = search.Views.GOV_CORP_INSTRUMENTS,
query = "Santander bonds",
top = 100,
filter = "IsPerpetualSecurity ne true and IsActive eq true and \
RCSIssuerCountryLeaf xeq 'United States'",
select = "MaturityDate, IssueDate, CouponRate, FaceOutstanding, FaceIssuedTotal, RCSCouponTypeLeaf",
order_by = "MaturityDate"
)
The above results look pretty good. However, I do see some odd entries for the 'FaceOutstanding' and 'FaceIssuedTotal' that may be undesirable for my purposes. As part of further investigation to better understand, I will demonstrate some possible ways to utilize the SearchPropertyExplorer features and how we may be able to clean up the above oddities.
SearchPropertyExplorer Features
The SearchPropertyExplorer was designed to facilitate many useful interrogation features enabling the user to rapidly discover properties. Given the power and flexibility of the Search service, you may need multiple ways to arrive at your discovery. For example, you may generally know, or guess, at the name of a property, or you may know a value to look for. If you are in a position where the name or value does not provide the answers, you may need to narrow down your results based on the property type or property that provides navigation. The bottom line is that the more tools you have available for use, the better the success rate at building your expressions.
Note: As you go through the exercise of locating properties, it's possible there may be no relevant candidates for you to choose from, despite all the tools available within the SearchPropertyExplorer. If you are unable to figure out why certain properties are unavailable for the kind of data you are retrieving, this may be related to the search criteria provided. Because the SearchPropertyExplorer only captures data for the first hit encountered, updating your search criteria may present a different data set and as a result, different properties. Otherwise, you may need to reach out to the LSEG Helpdesk and they can involve a Content Specialist who can investigate further.
Below is a general outline of some additional features. For each, I have provided some context when you may need to use them to give you a better understanding.
Properties
help(props.get_by_name)
Help on method get_by_name in module lseg.data.discovery._search_explorer._search_explorer_response:
get_by_name(name: str) -> lseg.data.discovery._search_explorer._properties.Properties method of lseg.data.discovery._search ...
Browse the properties names that have relative match with specified query. Results are represented
as the dataframe and dict of objects.
Parameters
----------
name: str
String to specify expected properties data.
Returns
-------
Properties
Examples
--------
>>> from lseg.data.discovery import SearchPropertyExplorer
>>> from lseg.data.content import search
>>> explorer = SearchPropertyExplorer()
>>> santander_bonds = explorer.get_properties_for(
... view=search.Views.GOV_CORP_INSTRUMENTS,
... query="santander bonds",
... filter="IsPerpetualSecurity ne true and IsActive eq true and not(AssetStatus in ('MAT' 'DC'))",
... )
>>> active = santander_bonds.get_by_name("active")
As demonstrated in our interrogation above, the get_by_name() provides a list of properties that match the expression text provided. We have taken full advantage of this capability to not only provide a list of candidates we can use to narrow down our result set, but allowing us to choose the output fields required.
Values
help(props.get_by_value)
Help on method get_by_value in module lseg.data.discovery._search_explorer._search_explorer_response:
get_by_value(value: Union[str, bool, int]) -> lseg.data.discovery._search_explorer._properties.Properties method of ...
Browse the properties example values that match the specified query. Results are represented
as the dataframe and dict of objects.
...
While the get_by_name() method is extremely useful, at times you may have an idea of the actual output values. For example, you may perform a query where you filter on a specific company but you don't know the property name associated with the company name. For example, in the above use case, we used the query expression "Santander bonds" to pull bonds for a specific company. In most cases, using a company name or ticker within a google-like query expression, will generate a reasonable list of bonds. However, there may be instances where Search will associate a specific query belonging to multiple organizations, or generate unrelated hits that may simply reference the company and thus generate an invalid list of bonds. This is entirely dependent on your query expression and that Search performs its queries that closely match your expression, as opposed to finding only hits related to the company expression you provided.
Alternatively, you may prefer to use the Company ID (Organization ID/Perm ID). In this scenario, when using LSEG Workspace to retrieve the Company Perm ID, eg: 8589934205, you can use this value to build out a filter that returns hits only related to this specific company ID.
# List all properties that contain the following value...
matches = props.get_by_value("8589934205")
matches.df
Using the above results, I can update my request by adding the following criteria within our filter:
"ParentOAPermID xeq '8589934205'"
# Generalize the query and update the filter expression
props = explorer.get_properties_for(
view = search.Views.GOV_CORP_INSTRUMENTS,
query = "bonds",
filter = "IsPerpetualSecurity ne true and IsActive eq true and " \
"RCSIssuerCountryLeaf xeq 'United States' and ParentOAPermID xeq '8589934205'"
)
props.hits_count
85
help(props.get_by_type)
Help on method get_by_type in module lseg.data.discovery._search_explorer._search_explorer_response:
get_by_type(property_type: Union[str, lseg.data.discovery._search_explorer._property_type.PropertyType]) ->
lseg.data.discovery._search_explorer._properties.Properties method of lseg.data.discovery._s...
Browse the types that match the specified query. Results are represented as the dataframe and dict of objects.
....
Where the get_by_name() and get_by_value() methods are unable to provide a clue for the information you are seeking, you can use the get_by_type() method to possibly help narrow down your discovery. For example, there may be a boolean condition that determines the applicability of the bond that may be important for your analysis. This mechanism does provide a way to narrow down your discovery. The following execution presents all properties that provide a conditional flag. Depending on your requirements, some of these conditions may be relevant to the details you require - you can include these within your filter expressions.
# Locate all properties that provide a boolean flag...
matches = props.get_by_type(PropertyType.Boolean)
matches.df
help(props.get_navigable)
Help on method get_navigable in module lseg.data.discovery._search_explorer._search_explorer_response:
get_navigable(prop: str = None, value: str = None) -> 'Properties' method of lseg.data.discovery._s ...
Browse all navigable properties, narrow down results by specifying name of navigable property. Results are
represented as the dataframe and dict of objects.
When you begin your journey to hunt down properties, you will notice a Navigable attribute as part of the output listed for each property. A value of true indicates that the property provides a well-defined bucket of values. For example, an industry sector, asset category, or country code is a typical navigator that defines a collection of values associated with that property. Navigators not only bucket data for users to understand the domain of how data is collected but include distribution of how many hits are specific to your search criteria.
Using navigators, I can discover an appropriate property.
As I noted above for our specific use case, some of the bonds returned do not contain values for "FaceOutstanding" or "FaceIssuedTotal". Doing some more investigated work with a content specialist, it was discovered that this data is not collected/available for bonds that belong to the "certificates of deposits" category. If I prefer to filter out bonds that belong to this category, I will need to determine a code or name related to a category that includes the "certificates of deposits" reference.
As a first step, let's determine if there is a navigable category property:
# Give me all navigators that may related to a category...
props = props.get_navigable('category')
props.df
At this moment, I can see many candidates to choose from. The one that seems most interesting/applicable is the 'RCSAssetCategory' and 'RCSAssetCategoryLeaf' properties. Let's list out all the asset categories for my latest query.
# Select the 'RCSAssetCategory' and 'RCSAssetCategoryLeaf' to navigate.
props = explorer.get_properties_for(
view = search.Views.GOV_CORP_INSTRUMENTS,
query = "bonds",
filter = "IsPerpetualSecurity ne true and IsActive eq true and " \
"RCSIssuerCountryLeaf xeq 'United States' and ParentOAPermID xeq '8589934205'",
navigators = "RCSAssetCategory, RCSAssetCategoryLeaf"
)
props.navigators
# Update our search to filter out 'Certificate of Deposit'. In our case, we'll use the corresponding category code: 'A:14'
# It's worth noting that I could simply ignore all categories except 'Bond' as an alternative.
# This depends on your requirements.
ld.discovery.search(
view = search.Views.GOV_CORP_INSTRUMENTS,
query = "bonds",
filter = "IsPerpetualSecurity ne true and IsActive eq true and " \
"RCSIssuerCountryLeaf xeq 'United States' and ParentOAPermID xeq '8589934205' and " \
"RCSAssetCategory ne 'A:14'",
top = 100,
select = "MaturityDate, IssueDate, CouponRate, FaceOutstanding, FaceIssuedTotal, RCSCouponTypeLeaf",
order_by = "MaturityDate"
)
Now that we appear to have a very good representation based on the requirements, I do have one final thought regarding the search for bonds. Using the 'query' option is usually reserved as a quick and easy way to retrieve a set of hits. While a great way to kick off the process, I typically do not rely on using 'query' within my final solution because there may be instances where certain unrelated hits unintentionally leak into the results. Sometimes it is difficult to really narrow down the total result set unless you really drill into specific properties and confirm relevancy. Instead, I typically rely on specific filters. For example, I could change the criteria to not include the 'query' specification and simply rely on the 'RCSAssetCategory' to narrow down as I'm only interested in bonds and preferred shares.
# Updating our search to remove the query but specifically specific interest in bonds and preferred shares.
ld.discovery.search(
view = search.Views.GOV_CORP_INSTRUMENTS,
filter = "IsPerpetualSecurity ne true and IsActive eq true and " \
"RCSIssuerCountryLeaf xeq 'United States' and ParentOAPermID xeq '8589934205' and " \
"RCSAssetCategory in ('A:J' 'A:K1')",
top = 100,
select = "MaturityDate, IssueDate, CouponRate, FaceOutstanding, FaceIssuedTotal, RCSCouponTypeLeaf, RCSAssetCategoryLeaf",
order_by = "MaturityDate"
)
While the result is the same when using the query, the moment you intend to try the above for many different companies, there may be unintentional hits leaking into the result set when using a query. The point being, there are multiple ways to arrive at your final result - something to be aware of.
Next Steps
The SearchPropertyExplorer interface is a simple value-add API you can access directly within the LSEG Data Library for Python to greatly accelerate your journey when building out your search criteria. I would recommend creating a simple test workbook utlizing this API to quickly perform many and multiple what-if scenarios. Because Search is so flexible and sensitive to many conditions, you may not realize the collection of hits does truly represent the data set you intended to retrieve. Not only does the SearchPropertyExplorer simplify the listing of the entire set of properties and conditions that may affect your search criteria, but the ability to hunt down whether these conditions play a role in the data set returned. The more you master Search and what this tool can provide, the greater the rate of success and the reduced amount of time spent.
