Publication Date

2021

Journal or Book Title

PLOS COMPUTATIONAL BIOLOGY

Abstract

For practical reasons, many forecasts of case, hospitalization, and death counts in the context of the current Coronavirus Disease 2019 (COVID-19) pandemic are issued in the form of central predictive intervals at various levels. This is also the case for the forecasts collected in the COVID-19 Forecast Hub (https://covid19forecasthub.org/). Forecast evaluation metrics like the logarithmic score, which has been applied in several infectious disease forecasting challenges, are then not available as they require full predictive distributions. This article provides an overview of how established methods for the evaluation of quantile and interval forecasts can be applied to epidemic forecasts in this format. Specifically, we discuss the computation and interpretation of the weighted interval score, which is a proper score that approximates the continuous ranked probability score. It can be interpreted as a generalization of the absolute error to probabilistic forecasts and allows for a decomposition into a measure of sharpness and penalties for over- and underprediction. Author summary During the COVID-19 pandemic, model-based probabilistic forecasts of case, hospitalization, and death numbers can help to improve situational awareness and guide public health interventions. The COVID-19 Forecast Hub (https://covid19forecasthub.org/) collects such forecasts from numerous national and international groups. Systematic and statistically sound evaluation of forecasts is an important prerequisite to revise and improve models and to combine different forecasts into ensemble predictions. We provide an intuitive introduction to scoring methods, which are suitable for the interval/quantile-based format used in the Forecast Hub, and compare them to other commonly used performance measures.

ISSN

1553-734X

ORCID

Gneiting, Tilmann/0000-0001-9397-3271; Bracher, Johannes/0000-0002-3777-1410; Reich, Nicholas/0000-0003-3503-9899

DOI

https://doi.org/10.1371/journal.pcbi.1008618

License

UMass Amherst Open Access Policy

Creative Commons License

Creative Commons Attribution 4.0 License
This work is licensed under a Creative Commons Attribution 4.0 License.

Funder

Helmholtz Foundation via the SIMCARD Information & Data Science Pilot Project; Klaus Tschira Foundation; US Centers for Disease Control and PreventionUnited States Department of Health & Human ServicesCenters for Disease Control & Prevention - USA [1U01IP001122]

Download

Share

COinS