Rise of the Data Connoisseur: Leveraging advanced weather forecasting for the clean energy transition

At every recent energy industry event, the highlight reel echoes the same three themes: the clean energy transition is underway, climate change and weather volatility are now constants, and AI is reshaping how we live and work while increasing energy demand.

Weather forecasting is at the intersection of these shifts, playing a pivotal role in the energy sector’s adaptation and evolution. In this article, we’ll explore how advanced forecasting is integral to these changes and what it unlocks for the industry.

As the clean energy transition accelerates — Europe is tracking towards sourcing 42.5 percent of its energy from power from wind, solar, and hydropower by 2030, and is aiming for climate neutrality by 2050 (source: European Environment Agency), with North America close behind — the grid has become more weather-dependent.

Historically, weather primarily governed demand; now it also governs supply. No one knows this better than the readers of this publication. Predicting solar insolation, cloud cover, 100m hub-height wind speeds, hydro flows, and precipitation is more crucial than ever. Accurate forecasts allow renewable asset decision-makers to effectively schedule maintenance, dispatch hydro, plan and estimate generation across portfolios, manage vegetation, hedge prices, model battery state of health curves, and more. In summary, reliable weather forecasts across all time horizons — real-time, day-ahead, S2S (subseasonal-to-seasonal), and long-term climate predictions — are essential for operating, planning, and expanding renewable energy assets.

The second shift — climate change accompanied by increased weather volatility —adds complexity to the forecasting challenge. Traditional climatological forecasting, which relies on historical averages, is becoming less relevant. Summers are generally hotter, and winters generally more erratic. As variability increases, rigorously calibrated probabilistic forecasts are increasingly critical, with attention to the tails of the distribution to understand extreme weather. This is where data best practices prove indispensable. For a deeper dive deeper, check out a recent US Climate and Variability (US CLIVAR) panel on data best practices in weather forecasting. 

In tandem with these shifts, AI and Machine Learning (ML) are unlocking new frontiers in weather forecasting. New foundational weather models, trained on massive datasets, are evolving to support diverse geospatial applications. Forecast emulators are rapidly improving their ability to forecast the forecast. On the S2S time horizon, purpose-built machine learning models are uncovering new, often non-linear teleconnections — involving variables like sea surface salinity —  along with more established ones. Across all timeframes, forecasting skill is improving at a remarkable pace.

As these industry dynamics unfold, new roles and skills are emerging. One of the most valuable yet challenging roles to leverage well is the data connoisseur.

Data connoisseur is not an actual title you can find on LinkedIn (Quantitative Analyst and Data Scientist are the actual job titles, among others) but rather a term that captures the qualities that these individuals exude. 

Data connoisseurs are experts, often with PhDs, who appreciate the meticulous care required to create excellent data. They value accuracy, reliability, and strong methodologies, fostering confidence in decision-making. These specialists are essential to effectively deploying data within large energy organizations, and they are increasingly in demand.

A recent study highlighted that commodity trading cash flows nearly doubled from 2018-2021, but many firms struggled to adapt to the growing challenges, including weather-driven fluctuations. The report cited outdated technology and a shortage of commercial talent skilled in data-driven methodologies as key barriers. Read: Data connoisseurs are in high demand. 

One critical aspect of a data connoisseur’s role is understanding and evaluating forecast accuracy and confidence. When forecasting the weather, transparent and data-driven providers measure forecast skill, offering metrics like Continuous Ranked Probability Score (CRPS), which evaluates both accuracy (how close the forecast is to the actual outcome) and precision (how confident the prediction is). This is akin to the Mean Average Error (MAE), but tailored to probabilistic forecasts.

Reliability is another hallmark of effective weather forecasting. For instance, if a forecast predicts a 60 percent chance of extreme drought but, historically, only 40 percent of those predictions materialized, the model is statistically overconfident. Adjusting probabilities based on real outcomes enhances reliability, enabling decision-makers to operate, plan, and develop energy assets with confidence.

Let’s consider an example: a probabilistic subseasonal solar insolation forecast for week 3, initiated in early June 2024, as summer heat is arriving. Several models, including NOAA’s GEFS, ECMWF, and private vendors, offer predictions at this horizon. A narrative-driven user might glance at the forecast maps (Figure 1) and then make decisions — power and gas trading decisions, solar asset management decisions, or price risk hedging. In contrast, a data connoisseur demands confidence in every pixel before taking action. They ingest the forecast via API, perhaps converting it from sun to power output using an SDK, and then integrate the probabilistic distribution — along with CRPS scores — into a decision model tailored to their specific thresholds and business objectives (Figure 2).

Figure 1: Narrative-driven forecast users review generalized weather forecast maps like this one for decision-making.

Figure 2: Data connoisseurs prioritize data accuracy and reliability, and value probabilistic forecasts with transparent skill scores for data-driven decision-making.

The takeaway? In the age of AI, data connoisseurs will thrive. Meteorologists who resist the advancements of AI and ML risk falling behind. Pairing reliable, calibrated forecasts with the right decision-support tooling is the key to navigating the clean energy transition amidst the unfolding effects of climate change.

Matt Stein is CEO and co-Founder of Salient. Salient’s expert forecasts and analytics provide the data and state-of-the-art methodologies that help industries work smarter.

Salient | www.salientpredictions.com