Proper Scoring Rules

A short post on scoring rules and their connection to a divergence metric.

A scoring rule is a metric that characterises the quality of a probabilistic forecast. If we are interested in forecasting rainfall, then we let the random variable $Y$ denote a future event and $\mathcal{Y}$ the set of all possible values that $Y$ could take. In this example, we would be define $\mathcal{Y}={\mathbb{R}}_{\ge 0}$ as it would not make sense to have negative rainfall. Our model is probabilistic and therefore outputs a probability distribution. We use $\mathcal{P}$ to denote the set of all valid probability distributions with support on $\mathcal{Y}$. When computing a scoring rule, we seek to compare our model’s forecasted distribution $p_t\in \mathcal{P}$ at time $t$ against a true observation $y_t$. In the context of our precipitation example, $p_t$ could be an exponential distribution with a rate parameter of 2 and $y_t$ would be a real value, such as 2.2mm, that would correspond to the true rainfall amount at time $t$.

A scoring rule $\mathcal{S}$ is then a function $S:\mathcal{P}\times \mathcal{Y}\to \mathbb{R}$. Lower scores are indicative of a higher quality forecast. If we let $q$ be the true probability distribution of rainfall, then for any $p\in \mathcal{P}$ we have $$\begin{array}{}\text{(1)}& S(p,q)={\mathbb{E}}_q[p,Y]=\int S(p,y)\mathrm{d}q(y).\end{array}$$ A scoring rule is defined as a proper scoring rule if for all $p$ $$\begin{array}{}\text{(2)}& S(q,q)\le S(p,q).\end{array}$$ Similarly, the proper scoring rule is strict if the inequality in (2) becomes a strictly less that. In this case, (2) will only achieve equality if $p=q$.

Scoring rules are a broad family of functions and there are many connections to statistical divergences. One example is the equivalence of the log-scoring rule $S(p,y)=\log p(y)$ and the Kullback-Leibler divergence (KLD) $\operatorname{KL}(q, p).Note the order of arguments in the KL-divergence operator matters as the divergence is asymmetric. To see this connection, we can write

$$\begin{array}{rl}S(q,q)-S(p,q)& =\int q(y)(\log q(y)-\log p(y))\mathrm{d}y\\ & =\int q(y)\frac{\log q(y)}{\log p(y)}\mathrm{d}y.\end{array}$$

We can see that starting from the log-scoring rule, in just two lines we've arrived at the definition of the KLD.