Turning Headlines into Signals: Using Structured News Metadata to Understand Energy Market Disruptions

Open the data session

The open_session() function establishes a data session to retrieve our content within the desktop.

Signal Definition

Before exploring the data, we define the structured codes that make up our energy supply disruption signal. Each code represents a classification dimension assigned to Reuters News stories — not a keyword, but a semantic label applied at publication time.

Centralizing these definitions here makes the signal explicit, easy to modify, and reusable across all queries in this notebook.

Signal query: (B:219) AND (I:6K OR M:3D0 OR M:2Z9)

Discovering Codes

But how were these codes identified? The newstokens helper provides a browse() method that lets you search within any taxonomy category using a server-side text query. For example, to find geographic codes related to the Middle East:

The browse() method supports several taxonomy categories — including BUSINESS_SECTORS, MARKETS, MORE_TOPICS, and others — and accepts a query parameter for server-side filtering. This makes it straightforward to explore the classification system and identify the codes you need for any signal definition. The codes used in this article were selected using exactly this kind of exploration.

1. The Problem with Keywords

The most intuitive way to find news about energy supply disruptions is to search for them by keyword. Let's try that first and see what comes back.

Story sources in keyword results:

source
social            35
webnews     8
reuters.com 4
newsroom   3

Social media stories: 35 of 50 (70%)

At first glance, many of these headlines look relevant. But look closer — the storyId URN itself reveals important context. Its structure — urn:{scheme}:{source}:{date}:{id}:{version} — encodes the source type of each story. Parsing it shows that a significant portion of keyword results come from social sources (tweets, retweets, commentary on X/Twitter) rather than editorial news content.

This is a key insight: keyword searches don't just match irrelevant topics — they pull in entirely different content types. Social media posts that happen to contain the right words are returned alongside editorial reporting, with no built-in way to distinguish them. The breakdown above quantifies this noise.

Beyond social media, keyword search may also pull in:

• Commentary and opinion pieces that mention disruptions in passing
• Market roundups where supply is one of several themes discussed
• Downstream stories (demand-side, retail pricing) that happen to use supply-related language
• Unrelated sectors where similar vocabulary appears (e.g., power grid outages, supply chain logistics)

The fundamental issue is that keywords match language, not meaning. A story can use the word "disruption" without being about a supply disruption event. And a story about a pipeline explosion may never use the word "disruption" at all.

To build a reliable signal, we need something that captures what a story is about — not just what words it contains.

2. Defining the Signal with Structured Metadata

Every Reuters News story is classified at publication time with a set of structured codes. These codes describe the story's topic, industry sector, named entities, geography, and more. They are assigned by a combination of editorial rules and automated classification — not by keyword matching.

This means we can define our signal not as a bag of words, but as a combination of structured attributes that together describe what we care about: energy supply disruptions.

Let's start by identifying the relevant building blocks.

The classification system organizes energy into meaningful sub-sectors — oil & gas, coal, renewables, nuclear, and so on. Rather than searching for the word "oil," we can target stories that have been classified as being about specific energy sub-sectors.

But sector alone is not enough. We also need to narrow by topic — specifically, supply-side disruption events.

Now we have two dimensions of classification:

By requiring both dimensions to be present, we can construct a much more precise filter than any keyword search can achieve. A story must be classified as energy-sector and classified as covering a supply-side event to pass through.

3. Retrieving the Signal

With our signal defined as an intersection of sector and topic classifications, we can now query for stories that match.

Stories matching structured signal: 50

Story sources in structured signal results:

source
reuters.com 36
webnews     9
newsroom   4
social            1

Social media stories: 1 of 50 (2%)

Compare these source breakdowns with the keyword results in Section 1. The difference is striking: keyword search returned a mix of social, webnews, and newsroom content — with social media making up a large share. The structured signal, by contrast, is dominated by editorial sources. Social media posts simply don't carry the classification codes that the structured query requires, so they're filtered out by design.

This single comparison — parsing the storyId URN from both approaches — tells most of the story. The structured query doesn't just find better topics; it finds a fundamentally different class of content. No additional metadata inspection is needed to see that.

4. What's Inside a Story: Examining the Metadata

To understand why this approach works, it helps to look at the classification attached to an individual story. Each story carries a subject array of structured codes — the editorial classification that describes sector, topic, geography, and more. We can extract those raw codes and resolve them into human-readable descriptions using our metadata helper.

Note: Not every story will have subject codes populated at the time you retrieve it. Breaking or high-priority stories are often published immediately and may initially arrive with an empty subject_codes array. Metadata such as subject classifications can be added or updated after the initial publication, at which point the story may be re-published with the enriched data. If the story you select has no subject codes, simply try a different index (e.g., iloc[1] instead of iloc[0]) or choose another story ID from the results.

A:4, G:3, G:A, G:AE, G:Q, G:V, M:3F, U:24, U:45, U:C, ...

You'll notice that some rows in the table above show a status of error with the message "News code identifier not found". These are consistently codes that begin with P: — for example, P:50554244391. These are not classification codes at all; they are PermIDs (Permanent Identifiers), LSEG's system for uniquely identifying real-world entities such as companies, people, and organizations.

PermIDs appear in the subject array because it serves as a unified tagging container — it carries not only the editorial classification (topic, sector, geography) but also entity annotations that link the story to specific real-world entities. This is by design: it allows consumers of the data to connect a story to the broader LSEG entity ecosystem, where a PermID can be used to look up company fundamentals, ownership structures, or other reference data. Because our metadata helper only resolves news classification codes, PermIDs fall outside its scope and can be safely ignored here.

The raw codes on their own are opaque. But once resolved through the metadata API, they become a structured profile of the story: which sector it covers, what topics it addresses, which geographies are relevant, and more.

This is the key insight: every story carries a machine-readable description of what it is about, not just the words it contains. That depth of classification is what allows us to build precise filters. And because it's applied consistently across all stories, the signal is repeatable — the same query run tomorrow will apply the same logic to new stories.

The newstokens helper also provides a convenience method — tokens.resolve_story(story_id) — that wraps the extraction and resolution steps above into a single call. We'll use it later when processing multiple stories at once.

5. Sharpening the Signal: Adding Geographic Focus

Our current signal captures energy supply stories globally. Depending on the use case, we may want to narrow further — for example, focusing on supply disruptions in a specific producing region.

Structured metadata makes this straightforward. We simply add a geographic dimension to our filter.

Middle East energy supply stories: 50

Each additional metadata dimension we layer in increases precision without sacrificing the systematic nature of the approach. We could further refine by:

• Sub-sector: Narrow to crude oil vs. natural gas vs. LNG
• Event type: Distinguish planned maintenance from unplanned outages
• Named entities: Focus on specific companies or facilities

The key insight is that these refinements are compositional — you build them by combining structured codes, not by inventing more complex keyword patterns.

6. Confirming the Difference Over a Common Window

The source breakdowns in Sections 1 and 3 already reveal the core difference: keyword search mixes social media, web aggregation, and editorial content indiscriminately, while the structured signal is inherently filtered to classified, editorial stories. To confirm this holds consistently, we can run both approaches over the same one-month window and compare their source composition side by side.

This also lets us check for blind spots — stories the structured signal captures that keyword search misses entirely, because the relevant events were described in language the keyword pattern doesn't anticipate.

The side-by-side source comparison confirms what Sections 1 and 3 already suggested: keyword search pulls in a large share of social media and aggregated web content, while the structured signal is composed almost entirely of editorially classified stories. The blind spots list further illustrates that keywords miss genuine supply events when the story uses different vocabulary.

This is the core value of the structured approach: it filters by meaning, not language — and in doing so, it eliminates entire categories of noise (social media, opinion, off-topic mentions) that keyword search cannot distinguish from relevant editorial reporting.

7. Measuring the Signal Over Time

Once we have a clean, well-defined signal, we can measure it. A simple but useful metric is headline frequency — how many disruption-related headlines are being published per day? Spikes in frequency often correspond to real-world events.

Spikes in the signal typically correspond to real-world disruption events — a refinery outage, sanctions escalation, weather-driven supply interruptions, or OPEC production decisions. The structured approach ensures these spikes reflect actual supply events, not just increased use of disruption-related vocabulary in broader market coverage.

8. From Signal to Context: What's Driving the Spike?

The chart above shows daily headline frequency over the coverage window. Some days have noticeably higher counts than others — these are the "spikes." But a spike on its own only tells us when disruption coverage intensified, not why.

To answer that, we take the peak day — the single date with the highest headline count — and decompose its metadata. By examining which classification codes appear most frequently across that day's headlines, we can identify the sub-sectors, geographies, and topics that drove the surge.

This metadata decomposition turns a quantitative spike into a qualitative narrative. Instead of just knowing that supply disruption coverage increased, we can see that it was driven by, for example, Middle East crude oil production stories — which immediately suggests what the market may be reacting to.

Because tokens.resolve_story() now accepts a list of story IDs and returns a DataFrame with story_id, the bulk decomposition step becomes a single call rather than a manual loop over stories.

9. Extending the Approach

Everything demonstrated here for energy supply disruptions is a pattern, not a one-off. The same methodology applies to any domain where Reuters News provides structured classification:

• Geopolitical risk: Sector + conflict/sanctions topic codes + geography
• Central bank policy: Topic codes for monetary policy + named entity codes for specific banks
• ESG events: Environmental or governance topic codes + sector + geography
• M&A activity: Deal/transaction topic codes + sector + named entities

In each case, the workflow is the same:

1. Define the signal as a combination of structured metadata dimensions
2. Retrieve stories matching the intersection
3. Measure the signal over time
4. Contextualize spikes using metadata decomposition
5. Combine with other data for deeper analysis

The structured metadata transforms news from a reading experience into an analytical building block — something that can be defined precisely, measured consistently, and integrated into broader workflows.

Summary: Keyword searches match language. Structured metadata matches meaning. By defining a supply disruption signal as an intersection of sector, topic, and geographic classification codes — rather than a pattern of words — we get a cleaner, more complete, and more repeatable view of what's happening in energy markets. That signal can then be measured over time and combined with other market data to support analytical workflows that go well beyond reading headlines.