Making single-proposition scoring rules

Accuracy scoring rules measure the value of your probability assignment’s closeness to the truth. A scoring rule for a single proposition p can be thought of as a pair of functions, T and F on the interval [0,1] where T(x) tells us the score for assigning x to p when p is true and F(x) tells us the score for assigning x to p when p is false. The scoring rule is proper provided that:

• xT(x) + (1−x)F(x) ≥ xT(y) + (1−x)F(y)

for all x and y. If you assign probability x to p, then xT(y) + (1−x)F(y) measures your expected value of the score for someone who assigns y. The propriety condition thus says that by your lights there isn’t a better probability to assign. After all, if there were, wouldn’t you assign it?

I’ve been playing with how to construct proper scoring rules for a single proposition, and I found two nice ways that are probably in the literature but I haven’t seen explicitly. First, let F be any monotone (not necessarily strictly) decreasing function on [0,1] that is finite except perhaps at 1. Then let:

• T_F(x) = F(1/2) − ((1−x)/x)F(x) − ∫_1/2^xu⁻²F(u)du.

I think we then have the following:

Fact 1: The pair (T_F,F) is a proper scoring rule.

Second, let T be any monotone increasing function on [0,1] that is finite except perhaps at 0. Let:

• F_T(x) = T(1/2) − (x/(1−x))T(x) + ∫_1/2^x(1/(1−u)²)T(u)du.

I think then we have the following:

Fact 2: The pair (T,F_T) a proper scoring rule.

In other words, to generate a proper scoring rule, we just need to choose one of the two functions making up the scoring rule, make sure it is monotone in the right direction, and then we can generate the other function.

Here’s a sketch of the proof of Fact 1. Note first that if F = c is constant, then T_F(x) = c − ((1−x)/x)c + c(x⁻¹−(1/2)⁻¹) = c + c − 2c = 0 for all x. Since the map F ↦ T_F is linear, it follows that if F and H differ by a constant, then T_F and T_H are the same. Thus subtracting a constant from F, we can assume without loss of generality that F is non-positive.

We can then approximate F by functions of the form ∑_ic_i1_[ai,1] with c_i non-positive (here I have to confess to not having checked all the details of the approximation) and by linearity we only need to check propriety for F =  − 1_[a,0]. If a = 0, then F is constant and T_F will be zero, and we will trivially have propriety. So suppose a > 0. Let T(x) =  − ((1−x)/x)F(x) − ∫₀^xu⁻²F(u)du. This differs by a constant from T_F, so (T_F,F) will be proper if and only if (T,F) is. Note that T(x) = 0 for x < a and for x ≥ a we have:

• T(x) = ((1−x)/x) + ∫_a^xu⁻²du = ((1−x)/x) − (x⁻¹−a⁻¹) = a⁻¹ − 1.

Thus, T = ((1−a)/a) ⋅ 1_[a,0]. Now let’s check if we have the propriety condition:

1. xT(y) + (1−x)F(y) ≤ xT(x) + (1−x)F(x).

Suppose first that x ≥ a. Then the right-hand-side is x(1−a)/a − (1−x). This is non-negative for x ≥ a, and the left-hand-side of (1) is zero if y < a, so we are done if y < a. Since T and F are constant on [a,1], the two sides of (1) are equal for y ≥ a.

Now suppose that x < a. Then the right-hand-side is zero. And the left-hand-side is zero unless y ≥ a. So suppose y ≥ a. Since T and F are constant on [a,1], we only need to check (1) at y = 1. At y = 1, the left-hand-side of (1) is x(1−a)/a − (1−x) ≤ 0 if x < a.

Fact 2 follows from Fact 1 together with the observation that (T,F) is proper if and only if (F*,T*) is proper, where T * (x) = T(1−x) and F * (x) = F(1−x).