Scoring rules for finitely additive probabilities on an infinite space

Michael Nielsen has drawn my attention to the interesting question of scoring rules for forecasts on an infinite sample space Ω. Suppose that the forecasts are finitely additive probabilities on Ω, and that a scoring rule assigns an inaccuracy score s(p) to every finitely additive probability p on Ω (and maybe also to some or all inconsistent credences, but that won’t matter to us), where s(p) is a function from Ω to [ − ∞, ∞]. If we think of Ω as a space of situations or worlds, then s(p)(ω) measures how far off your forecast p would be if you were actually in situation or world ω.

Say that two forecasts p and q are orthogonal provided that there is a subset A of Ω such that p(A)=1 and q(A)=0. Orthogonal forecasts disagree as badly as possible.

Here is a plausible condition on a scoring rule s:

• A scoring rule s has orthogonal fine grain if for any two orthogonal forecasts p and q, there is an ω ∈ Ω such that s(p)(ω)≠s(q)(ω).

Requiring orthogonal fine grain is plausible, because if forecasts disagree maximally, we would expect them to score differently in at least one situation.

Theorem: Assume the Axiom of Choice. If Ω is infinite, then no scoring rule has orthogonal fine grain. In fact, for any scoring rule s there will be orthogonal maximally opinionated forecasts p and q such that s(p)=s(q) everywhere on Ω.

Here, a forecast p is maximally opinionated provided that for any event A, p(A) is either 0 or 1. Note that the theorem does not assume any continuity or propriety.

An immediate corollary is that there is no strictly proper scoring rule (i.e., scoring rule s such that E_p(s(p)) < E_p(s(q)) for every distinct pair of forecasts p and q) for an infinite sample space, a result Michael Nielsen communicated to me under additional assumptions on s. This in turn should either make one a little suspicious of arguments for probabilistic consistency in finite cases that are based on an insistence on strict propriety, or it should push one in the direction of requiring countable additivity in the infinite case.

Proof of Theorem: Any two maximally opinionated forecasts are orthogonal. For suppose that p and q are maximally opinionated and not identical. Then for some A ⊆ Ω we have p(A)≠q(A). Thus, either p(A)=1 and q(A)=0 or p(Ω − A)=1 and q(Ω − B)=0, and so we have orthogonality.

The set of maximally opinionated forecasts is in one-to-one correspondence with the set of ultrafilters on Ω which has cardinality 2^2|Ω| (Proposition 6 here), of course assuming the Axiom of Choice (without which all the ultrafilters might be principal, and hence there might be only |Ω| of them).

On the other hand, s(p) for every p is a function from Ω to [ − ∞, ∞]. The cardinality of the set of such functions is (2^ℵ₀)^|Ω| = 2^ℵ₀×|Ω| = 2^|Ω| for infinite Ω, assuming the Axiom of Choice. Hence, there are more maximally opinionated forecasts than possible scores, and hence some maximally opinionated forecasts must share the same score. But we saw that any two maximally opinionated forecasts are orthogonal. QED