Traditional weather forecasting relies on numerical models with decades of refinement, evaluated using well-established tools designed for gridded data (like GRIB or NetCDF) and fixed forecast horizons. But machine learning is changing the game - offering unstructured, probabilistic, and dynamically adapted weather predictions. Existing evaluation frameworks assume rigid data formats and workflows, and often remain closed-source, limiting collaboration.

In contrast, the WeatherGenerator project is a multi-institutional initiative that operates on data fundamentally different from traditional gridded formats. It ingests and produces unstructured data and operates on irregular meshes. Furthermore, its machine learning development cycles require rapid experimentation, short feedback loops, and frequent model iteration. Evaluation frameworks that require extensive preprocessing, rigid data alignment, or complex operational dependencies are therefore poorly suited to this context.

This led to the recognition that existing tools were insufficiently flexible for the project’s objectives. In particular, the rigidity of traditional evaluation frameworks made them unsuitable for fast turnaround evaluation during early development phases, when models and data streams are frequently changing.

In response to these challenges, the WeatherGenerator consortium developed a new evaluation framework known as WeatherGenerator FastEvaluation package. The primary objective of this framework is to provide a flexible, open-source, and extensible evaluation solution tailored to the needs of machine learning–based weather and climate modelling. Rather than attempting to replicate the full functionality of long-established operational verification systems, the tool focuses on enabling rapid, consistent, and reproducible evaluation across a wide range of data types and experimental configurations.

A central design principle of the WeatherGenerator FastEvaluation tool is the minimisation of assumptions about the structure and format of the data being evaluated. Unlike many existing tools, it does not assume that inputs are provided on a regular latitude–longitude grid, nor does it require fixed forecast lead times or a single deterministic output stream.

One of the main features of the tool is its ability to perform scoring and visualisation within a unified framework. Rather than treating metric computation and plotting as separate tasks, the framework allows users to compute skill scores, generate two-dimensional maps, produce histograms, and create summary visualisations in a consistent and coordinated manner. This integration simplifies the evaluation workflow and reduces the potential for inconsistencies between numerical results and visual diagnostics.

From a performance perspective, WeGen FastEvaluation supports parallel processing of data, allowing computationally intensive score calculations to be distributed across available resources. This capability is essential for evaluating large datasets or ensemble forecasts on high-performance computing systems.

Model performance across forecast lead times can be assessed using line charts, which illustrate the evolution of selected metrics. For ensemble forecasts, these plots can also represent ensemble spread, enabling probabilistic evaluation and uncertainty quantification.

For comparative analysis across different trained models, the package includes a set of additional tools specifically designed for this purpose. Score cards, bar plots, ratio plots and heat maps summarize relative performance changes of the models with respect to a chosen baseline across variables of interest.

The evaluation package also includes an export tool to connect the WeatherGenerator output to the format required by the other tools and aid more comprehensive analyses. This allows users to convert native WeatherGenerator outputs, saved during the inference of a model in Zarr format, to a variety of widely accepted file formats e.g. NetCDFs and GRIBs, that can be read by external open source and national meteorological services and partner institutions’ in-house tools. Furthermore, this tool provides the infrastructure for model engagement upon a more widely available public release.

At the moment the export tool supports conversions to CF-compliant netCDF and GRIB formats suitable for the UKMet Office CSET evaluation package, the METNorway verification and Diana tools, as well as the ECMWF’s Quaver and the MeteoSwiss mlEval tools. A collaborative effort to create an extension to existing export tool resulted in performance improvements for the existing code infrastructure and provides an example upon which other national meteorological services can draw upon to devise their own custom outputs.

The tool also supports re-griddings to other regular grids at conversion level through the ECMWF library earthkit-regrid. With this at the user’s disposal, WeatherGenerator’s results can be more easily compared with traditional NWP models like IFS and recent ML models like Microsoft’s Aurora, supporting both model development and evaluation. We will support more file formats as user requirements evolve and support efforts to evaluate the WeatherGenerator model using existing tools.

In summary, WeatherGenerator FastEvaluation is one of the first examples of flexible, scalable, and open-source solution that bridges the gap between traditional verification tools and the needs of modern machine learning workflows, enabling efficient, reproducible, and comprehensive model evaluation for these new set of multi-resolution and multi-purpose models.