Let s be an accuracy scoring rule on a finite probability space. Thus, s(P) is a random variable measuring how close a probability assignment P is to the truth. Here are two reasonable conditions on the rule (the name for the second is made up):
Propriety: EPs(P)≤EPs(Q) for any distinct probability assignments P and Q.
Investigativeness: EPs(P)≤P(A)EPAs(PA)+P(Ac)EPAcs(PAc) whenever 0 < P(A)<1.
where EP is expected value with respect to P, PA is short for P(⋅|A), and Ac is the complement of A. Propriety says that if we are trying to maximize expected accuracy, we will never have reason to evidencelessly switch to a different credence. Investigativeness says that expected accuracy maximization never requires one to close one’s eyes to evidence because the expected accuracy after conditionalizing on learning whether A holds is at least as good as the currently expected accuracy. And we have strict versions of the two conditions provided the inequalities are always strict.
It is well-known that propriety implies investigativeness, and ditto for the strict variants.
One might guess that the other direction holds as well: that investigativeness implies propriety. But (perhaps surprisingly) not! In fact, strict investigativeness does not imply propriety.
Let s(P) be the following score: s(P)(w)=|{A : P(A)=1 and w ∈ A}|. In other words, s(P) measures how many true propositions P assigns probability 1 to. It is easy to see that s(PA)≥s(P) everywhere on A, and ditto for Ac in place of A, so the right-hand side in (2) is at least as big as P(A)EPAs(P)+P(Ac)EPAcs(P)=EPs(P).
But propriety does not hold as long as our probability space has at least two points. For let P be any regular probability—one that assigns a non-zero value to every non-empty set—and let Q be any probability concentrated at one point w0. Then s(P)=1 everywhere (the only subset P assigns probability 1 to is the whole space) while EPs(Q)≥1 + P({w0}) > 1 (since Q assigns probability 1 to {w} and to the whole space), and so we don’t have propriety.
If we want strict investigativeness, just replace s with s + ϵs′ where s′ is a Brier score and ϵ is small and positive. Then we will have strict investigativeness for s′, and hence for s + ϵs′ as well, but if ϵ is sufficiently small, we won’t have propriety.
It is interesting to think if investigativeness plus some additional plausible condition might imply propriety. A very plausible further condition is that if P is at least as close to the truth as Q for every event, then P gets a no-worse score. Another plausible condition is additivity. But my examples satisfy both conditions. I don’t see other plausible conditions to add, besides propriety as such.
No comments:
Post a Comment