Speaker – Lenny Smith, Professor, Virginia Tech. Hosted by Walt Robinson. 

Title – Reflection versus Regurgitation: Simulation, Statistics, and Insight from Fitzroy to ECMWF’s AIFS. (link to relevant paper)

Abstract – There is considerable dispute today over when scientific forecasting is best grounded in sophisticated regurgitation of past observations, and when it is best grounded in reflection on the dynamics thought to govern the system. Disputes over forecast quality are not merely disputes about skill; they are disputes about what counts as evidence, what constraints matter, what timescales are considered and who bears the consequences of error.  Personal choices are often defended with almost religious zeal, and the interpretation of performance statistics is sometimes biased by personal circumstance. Conflicts between Fitzroy, Galton and the British Board of Trade demonstrate that the disagreements go well beyond scientific inference. As in the Singular Vectors Wars, today’s discussion of the appropriate role of narrow AI (nAI) in weather forecasting sometimes falls out of focus and into oversell. It is argued that nAI is best used not as an oracle replacing physical simulation, but as a punching bag to address questions that could not be asked previously.

Forecasts are best placed into the form P(x| I), the probability of a future outcome x conditioned on the information available when it is cast, along with an quantitative estimate of the forecast’s irrelevance. It is argued that Jack Good’s log score (Ignorance) is the rational choice of metric here. The choice of metric is never neutral, particularly when it is made post hoc. Historically, predicting how skill will evolve as I evolves is fraught with misadventure.

A distinction is then drawn between two broad approaches to forecasting. Simulation-based forecasts encode structural assumptions about the target system, grounded in first principles as currently understood. Empirical-Statistical approaches rely more heavily on the observational archive couched in a generic mathematical structure. The skill of each depends on the information in the observational archive, as does the ability to evaluate their relative skill.  The line between these two approaches is not sharp; questions here include not only how to obtain the most informative forecasts today,   but also how forecast skill is likely to improve with time as observations accumulate, understanding deepens and compute power evolves. The costs of succumbing to convenient rhetoric, snake oil and vaporware promises, or mistakenly assuming that truly superior skill today implies superior skill in the future are noted.

Forecasts are not judged on epistemic grounds alone; they are also embedded in institutions, exposed to political pressures, and evaluated in the light of actions, consequences, and blame. The way forecasts are formulated and distributed has impacts far beyond the time scale of the forecast system itself. Where the underlying system can be understood scientifically,  simulation forecasting supported by an nAI inquisition is to be favored as long as scientific understanding is advancing.

Speaker bio – Professor Leonard A Smith received his PhD in Physics at Columbia University. From 1992 to 2020 he held a Senior Research Fellowship (mathematics) at Pembroke College, Oxford (UK); he was Professor of Statistics and director of the LSE’s Centre for the Analysis of Time Series for two decades. In 2020 he joined Virginia Tech as a Professor of Engineering. He is active in the formation, initialization, interpretation, evaluation and attempted application of probability forecasts in many areas, holding a general interest in improving simulation modelling and scientific support for decision making, with aspirations of relevant uncertainty quantification and the escape from model land. Professor Smith was awarded a Selby Fellowship from the Australian Academy of Sciences, his bestselling book A Very Short Introduction to Chaos is now available in over eight languages; he was active in the original experimental design(s) of climateprediction.net and  the World Meteorological Organization’s THORPEX Science Plan. He holds the Royal Meteorological Society’s Fitzroy Prize , and shared both a AAAS Science Journalism Gold Award for Television and a European Science TV and New Media Award for Best Presentation of Science on an Environment Issue  as science consultant for the BBC’s “Climate Change by Numbers“, and is an AGU Charney Lecturer.