Deriving four winter weather regimes with Self-Organising Maps (SOM)
Summary. Self-Organising Maps (SOM) are a simple neural-network method used here to extract winter weather regimes by grouping daily circulation patterns over the Euro-Atlantic region. The approach is straightforward to apply, but results are sensitive to a few key choices (domain, variables, normalisation, map size, and training parameters). Below we explain how SOM work, and apply it to the identification of the four winter weather regimes.
Why weather regimes matter
Weather regimes are recurrent, persistent large-scale flow patterns. They evolve more slowly than day-to-day weather, which makes them carry predictive value at week-ahead timescales and tie directly to surface impacts, such as renewable energy generation or electricity demand (refer to [1] for more details). A general understanding of these regimes can, for instance, help with the planning and management of electricity system.
What are Self-Organising Maps?
Self-Organising Maps (SOM, [2]) are constituted by a 2D grid of “neurons”. Each of these neurons is initialised with a vector of the same dimension as the input (in our case, the 500 hPa geopotential height anomaly). Then, for each input pattern the SOM is trained by following two steps:
- Find the Best Matching Unit (BMU): the BMU is the neuron which is closest to x (usually calculated through the Euclidean distance).
- Update the BMU and its neighbours: all neurons are updated based on the new sample considered. How much they are updated depends on two factors: the learning rate, and the distance to the BMU. The update is strongest for the BMU, and other neighbours are updated depending on their distance to it.
This process is repeated over the entire dataset, where the neurons become more representative of the data and nearby neurons represent similar patterns, preserving topology. After training, the stardard weather regime treatment can be applied:
- A weather regime through the time series of BMUs
- The patterns for weather regimes are identified from the mean of all fields assigned to a BMU
Important note: The specifics of the configuration of the SOM are inherent to the specific result obtained. Parameters such as grid size (in our case, 2×2 to target four regimes), initialisation (random vs. PCA), neighbourhood radius and how it decreases with iterations, learning rate and how it decreases with iterations, and total iterations. Contrarily to other algorithms, increasing the number of iterations does not improve the SOM results, and it actually leads to overfitting. The specific setup for all of these parameters is key to the final result. For further information on the application of SOM, refer to [3].
Data and domain
- Dataset: ERA5 reanalysis
- Variable: 500 hPa geopotential height (z500)
- Region: 90°W–30°E, 20°–80°N
- Period: Winters 1991–2020 (daily means)
- Grid: 1° spacing (natively downloaded)
Pre-processing
- Area weighting: multiply each grid point by the cosine of its to balance area toward the poles.
- Anomalies: subtract the winter-mean at each grid point.
- Reshape: stack the 2-D field to a 1-D vector per day (as per algorithm requirements).
- Standardise: z-score each grid point over time (to avoid biases in SOM).
SOM setup
When setting up the SOM, a few parameters need to be chosen. The size of the SOM is determined by how many neurons (equivalent to clusters in this case) we are aiming to get (four for the winter weather regimes). However, many other parameters determine the specifics of the SOM. Here is an overview, and our selection for each:
- sigma - initial nighbourhood radius, it decays to 0 with iterations
- learning rate - rate at which the neurons are updated, decays to 0 with iterations
- neighbourhood function - how the influence of a sample decays with the distance from its BMU
- number of iterations - total number of iterations, in our case 5000 iterations account for roughly three passes of the whole dataset
Note that a random seed was used to ensure that results can be reproduced.
Code: from ERA5 to a trained 2×2 SOM
The code in this example is structured for use with the minimal SOM package MiniSOM [4].
import xarray as xr
import numpy as np
import pandas as pd
from minisom import MiniSom
from sklearn.preprocessing import StandardScaler
# Assume da_z500 contains the winter data for z500 (time, latitude, longitude) for the entire domain and subtract the anomaly
da_anom = da_z500 - da_z500.mean(dim='time')
# Calculate area weighting
da_weighted = da_anom * np.cos(np.radians(da_anom['latitude']))
# Reshape the data to [samples, features]
X = da_weighted.stack(points=("latitude","longitude")).transpose("time","points").values
# Standardise features across time
scaler = StandardScaler(with_mean=True, with_std=True)
X_std = scaler.fit_transform(X)
# Train 2x2 SOM (four regimes)
som = MiniSom(x=2, y=2, input_len=X_std.shape[1], sigma=1.0, learning_rate=0.5,
neighborhood_function='gaussian', random_seed=42)
som.random_weights_init(X_std)
som.train_random(X_std, num_iteration=5000)
# Assign each day to its BMU
bmus = np.array([som.winner(x) for x in X_std]) # array of (i,j) pairs
labels = np.ravel_multi_index((bmus[:,0], bmus[:,1]), dims=(2,2)) # 0..3
# The average field of each BMU can be recovered from the mean of the fields with each label
Alternative workflow: large SOM for mapping + k-means to 4 regimes
A more common use for SOM is to use it as a dimensionality reduction tool that maintains topology. Therefore, something that has been done in the literature is to run a large SOM (such as 10×10 or 20×20, see [5] for an example) to discretise the state space. Then, the neurons are used as input to k-means to get four clusters. Each day inherits its regime via its BMU’s cluster.
Some additions to the code are required for this application:
from sklearn.cluster import KMeans
# Train 10x10 SOM
som_big = MiniSom(x=10, y=10, input_len=X_std.shape[1], sigma=1.0, learning_rate=0.5,
neighborhood_function='gaussian', random_seed=42)
som_big.random_weights_init(X_std)
som_big.train_random(X_std, num_iteration=10000)
# Collect neuron codebook vectors and cluster to 4 regimes
codebook = som_big.get_weights().reshape(100, -1)
km = KMeans(n_clusters=4, random_state=42, n_init="auto").fit(codebook)
node_regime = km.labels_.reshape(10,10)
# Day-level labels: regime of each day’s BMU
bmus_big = np.array([som_big.winner(x) for x in X_std])
labels4 = np.array([node_regime[i,j] for i,j in bmus_big])
# Composites by regime (as before)
regime_composites_km = []
for r in range(4):
comp_w = anom_fields[labels4==r].mean(axis=0)
comp = comp_w / w.values[:,None]
regime_composites_km.append(comp)
Results at a glance
Both approaches are able to recover the four Euro-Atlantic winter weather regimes, at least to some extent. Exact spatial detail and regime frequencies depend on the training setup and analysis window. By construction, the approach combining SOM and k-means presents results closer to the standard method using PCA in combination with the clustering.
 |
| Figure 1: 500 hPa geopotential height anomalies representative of the four winter weather regimes as obtained from the direct results of the SOM. The names assigned to regimes respond to their similarity to the four traditional winter weather regimes. |
 |
| Figure 2: 500 hPa geopotential height anomalies representative of the four winter weather regimes as obtained from the application of a 10 by 10 SOM, followed by k-means clustering. |
Using SOM in conjunction with the k-means clustering leads to very similar results to the standard winter weather regimes (https://conorsweeneyucd.blogspot.com/2025/10/deriving-four-winter-weather-regimes.html). However, using only SOM leads to well defined NAO Negative and Scandinavian Blocking regimes, but to some confusion for the NAO Positive and Atlantic Ridge regimes, none of which show clear signal in the weather regimes derived from just SOM.
Sensitivity and limitations
- No guaranteed optimum: increasing iterations does not ensure “the” answer; too many can overfit to single days.
- Hyperparameters matter: grid size, neighbourhood radius and its change function, learning rate and its change function, or initialisation, among others, can change the results of the SOM.
- Domain choices matter: spatial, season definition (e.g., DJF vs. NDJFM), temporal resolution (daily vs 6-hourly or 3-hourly), variable (z500 vs. MSLP), and detrending choices can all shift patterns and frequencies.
Practical pointers
- Run short sensitivity sweeps (e.g., seeds, iterations ±20%, radius schedule, or even larger SOM networks) to compare regime stability.
- Validate on sub-periods (e.g., 1991–2005 vs. 2006–2020) to confirm persistence.
- Keep daily labels to study links to relevant surface-level features
Reproducibility checklist
- Random seeds are fixed for initialisation and k-means.
- Training parameters are tracked (iterations, training rate and its decay, neighbourhood function and its decay).
- Domain, variable, season definition, and anomaly/climatology window are registered.
- Exact ERA5 file versions and preprocessing steps are taken.
References
- How to make use of weather regimes in extended-range predictions for Europe (ECMWF Newsletter 165 - Autumn 2020).
- T. Kohonen (2001). Self-Organizing Maps, Springer. https://doi.org/10.1007/978-3-642-56927-2
- Self-Organizing Maps (DataCamp tutorial).
- MiniSom (simple Python SOM implementation).
- Ohba, M., Kanno, Y., & Nohara, D. (2022). Climatology of dark doldrums in Japan. Renewable and Sustainable Energy Reviews, 155, 111927. https://doi.org/10.1016/j.rser.2021.111927
