Globally, there are more than a dozen seasonal climate models from international meteorological agencies providing rainfall and temperature predictions up to six months into the future. These models are all slightly different and have varying accuracy and consistency depending on where you are farming.

This crucial requirement of climate forecasts to provide accurate and consistent climate predictions is referred to as its ‘skill’. For growers and advisers, this comes down to knowing what is the most accurate climate model for their region, if they are to act upon a forecast with any confidence.

Forecast skill is typically measured in relation to the extent to which it can provide a better prediction than one based on chance, or any potential outcome that has been observed previously for, say, the past 30 years.

The skill of a model is quantified on past model performance – instances where the forecast and the corresponding observed outcome can be verified. This measure can be undertaken for specific regions and times of year where users of the forecast may need climate information for particular purposes.

To improve cropping applications, a GRDC-invested project dubbed ‘AgScore’ is assessing the skill of seasonal climate forecasting systems and their value to grain growing decision-making.

AgScore is being led by CSIRO, which is partnering with Weather Intelligence, a company that provides climate analytics services and research for the mining, energy and agricultural industries in Australia.

The project is being co-funded under the Managing Climate Variability (MCV) program, a collaboration between GRDC, Meat & Livestock Australia, AgriFutures Australia, Cotton RDC and Sugar Research Australia Ltd. It started in July 2020 and will conclude in December 2021.

Project at large

“The broad AgScore project has three work packages and fundamental to them all is the use of an innovative software tool called AgScore™,” project leader Dr Patrick Mitchell says.

“The AgScore™ tool provides a robust approach to benchmarking the performance of seasonal climate models using agriculturally relevant metrics.”

The first work package will test a group of 10 models that currently produce operational forecasts for Australia, including systems such as NASA’s GEOS-S2S-2 model and the UK Met Office’s GloSea5 model, as well as the Bureau of Meteorology’s ACCESS-S1 system.

This work package will compare how well these models forecast rainfall and temperature across broad agricultural regions in Australia. It will determine where and when models might offer skillful forecasts for farmers.

The second work package takes some of the better-performing models and investigates their skill in simulating agricultural productivity using the AgScore™ tool. AgScore™ has been developed by CSIRO to evaluate forecast performance specifically within an agricultural context.

“Rather than just asking ‘How well does this model forecast rainfall?’, AgScore™ also asks ‘How useful is this climate model in helping to forecast grain yield?’,” Dr Mitchell says.

“Under the hood, AgScore™ is a cloud-based tool which executes the agricultural systems models, such as Agricultural Production Systems sIMulator (APSIM), for a set of representative locations across the grainbelt.”

Dr Mitchell says the tool is aimed at climate and agricultural scientists who might not have expertise in complex crop simulation modelling and need a consistent method for comparing climate model output.

AgScore™ users upload a climate forecast dataset and the tool executes and analyses thousands of crop simulations and compares climate model-based yield results against yield results generated from a baseline climatology forecast over the same period.

AgScore™ uses these results to produce interactive report cards for different agricultural industries for a particular climate model. The AgScore™ tool evaluates the potential biases and weaknesses of a climate model for predicting agriculturally relevant metrics and distils this information into summary scores for different metrics.

Finally, work package three will assess the economic value of forecasts for some key on-farm decisions in grain production using a number of grower case studies. This component of the project addresses the need to understand how the use of seasonal forecasts can assist strategic decisions, such as crop selection and fertiliser management, and provide improved economic outcomes for the farm business.

Timeline for delivery

Work package one will be delivered in early 2021.

“It is likely that there are certain times within the year where a subset of the models evaluated have higher skill and can offer improved outcomes for on-farm decisions than others,” Dr Mitchell says.

“The regional analysis provided in the first work package will also enable us to identify the better-performing models for each region and remove those that have very low skill from subsequent analyses in the project.”

The results will be communicated at different levels to growers, advisers and extension specialists in climate risk management, as well as the broader climate science community.

‘Report cards’ generated from the AgScore™ tool in work package two are anticipated for a mid-2021 delivery (Figure 1). These report cards will provide a summary of the important aspects of model performance for predicting agricultural productivity. They will provide diagnostic information that gives climate and agricultural scientists important feedback as to why a particular model performed well or poorly. They can use this information to target future effort and investment towards improving particular components of the seasonal climate models and their delivery as forecasts.

Figure 1: Example of a potential AgScore™ report card. A user submits climate forecast data and AgScore™ runs a suite of wheat simulations using observed and modelled data in a cloud-based computing platform (Senaps) developed by CSIRO. The tool computes skill metrics from thousands of simulations and returns the results of the analysis as an interactive dashboard.

Source: CSIRO

Work package three will be delivered in late 2021 and will produce a technical report as well as plain English su mmaries of the results from the on-farm case studies.

It is hoped the AgScore project will inform the usage of various climate models by climate forecast providers in a range of agricultural decision support tools that rely on seasonal climate data such as the newly launched ‘Farming Forecaster’ (https://farmingforecaster.com.au/).

Seasonal climate outlook services such as Agriculture Victoria’s ‘The Break’ will be strengthened by having a common benchmark for the models they draw on to provide an outlook on the season ahead. Ultimately, AgScore will help communications and extension specialists remove some of the ‘noise’ associated with different climate forecast information and ensure growers get access to the best-in-class seasonal climate information for their enterprise and region.

More information: Dr Patrick Mitchell, patrick.mitchell@csiro.au, 0459 813 793