Let P be the set of countably additive probabilities on a countable set Ω. A strictly proper accuracy scoring rule on some set of credences C that includes P is a function s from C to [ − ∞, M]Ω for some finite M such that Eps(p)>Eps(q) for any p ∈ P and any q ∈ C: i.e., from the point of view of each probability in P, that probability has the highest expected accuracy. (It’s a little easier for my example to work with accuracy scoring rules.)
We can identify P with the non-negative functions of ℓ1(Ω) that sum to 1. This identification induces a topology on P (based on the norm and weak topologies on ℓ1(Ω) which are the same). On [ − ∞, M]Ω we will take the product topology.
We say that a scoring rule is uniformly bounded provided there is some finite R such that |s(p)(ω)| < R for all p ∈ C and ω ∈ Ω. We say that one function strictly dominates another provided that the former is strictly less than the latter everywhere.
Theorem: There is a uniformly bounded strictly proper scoring rule s on C that is continuous on P and a credence c ∈ C − P such that s(c) is not strictly dominated by s(p) for any p ∈ P.
In a recent post, I showed that it’s not possible to use strictly proper scoring rules to produce a strict domination argument for probabilism (the thesis that all credences should be probabilities) in infinite cases when we take probabilities to be finitely additive, because in the finitely additive case there is no strictly proper scoring rule. The above Theorem is a complement for the countably additive case: it shows that there are nice strictly proper scoring rules for the countably additive probabilities, but they don’t support the strict domination results that arguments for probabilism seem to require.
There is a remaining open question as to what happens when one further assumes additivity of the scoring rule. But I do not think additivity of a scoring rule is a reasonable constraint, because it seems to me that epistemic utilities will depend on some global features of a forecast.
Here is a sketch of the proof of the theorem. Assume Ω is the natural numbers. Identify P with the non-negative members of ℓ1 that sum to 1. Let s(p)(n)=p(n)/∥p∥2 for p ∈ P. Note that the ℓ2-norm is continuous on ℓ1, and hence s is continuous on Ω. Observe that s(p)∈ℓ1. By the Cauchy-Schwarz inequality (together with the condition for equality in it), s is strictly proper on P. Define s(c)(n)=1/2(n + 1) for any credence c that is not in P. Note that Eps(p)=∥p∥2 for all p. Observe that Eps(c)<∥s(c)∥2∥p∥2 for every p by Cauchy-Schwarz again. But ∥s(c)∥2<1. Thus, s is strictly proper. But s(c) for c not a probability is not a member of ℓ1, and hence is not dominated by any score of a probability.
Typo? 4th last line: … for any credence c that is not in C. (should be P?)
ReplyDeleteThe scoring rule s is not as ‘nice’ one could wish for, at least if I understand it correctly. It is continuous on P, but not continuous around P. This seems like ‘cheating’. At a minimum, it is intuitively unsatisfying.
Is the scoring rule proper? Example (if I’m thinking straight):
ReplyDeletep = 1 , 0 , …
q = 2/3, 1/3, 0 , …
s(q) = (2/3, 1/3, 0 , …)/(4/9 + 1/9) = (2/3, 1/3, …)/(5/9)
EpS(q) = (1 * (2/3) + 0 * (1/3))/(5/9) = 6/5 > 1.
Yup: I got the denominator in the definition of s(p) wrong. I fixed the rest up and I am thinking it's OK now. But I haven't been sleeping well and so it could be all confused.
ReplyDeleteAs for lack of continuity around P, it's not clear how to define continuity outside of P.
Here's something that doesn't quite do the task, as it's not defined on all of C. But it will be defined on a pretty large set of credences C. Let C* be all the credences c in C such that for some n, 0<sum |c{n}|<=1.
Let z0 be (1/2,1/4,1/6,1/8,...) (anything with l2-norm less than one and infinite l1-norm will do).
Given a credence function c, let
D(c) = max(1,2|c{0,1}-c{0}-c{1})|/(1+|c{0,1}|+|c{0}|+|c{1}|)).
Intuitively, D is a continuous function given a reasonable topology on C. Note that D(p)=0 for any probability p, but there is a c0 in C* such that D(c0)=1 (e.g., let c0{0,1}=1, c0{n}=0 for all n other than 2, c0{2}=1).
Let s0(c)(n) = c{n}/sqrt(c{0}^2+c{1}^2+...} for any credence c. This is strictly proper on C*.
Let s(c)=(1-D(c))s0(c)+D(c)z0. This should be continuous given a reasonable topology on C*. It should be strictly proper on C*. And on P it coinicdes with s0. But s(c0)=z0.
There is probably some way to extend tweak this construction so it works on all of C.
The new version seems fine. I will have to think about the continuity tweak.
ReplyDeleteMy s0 score is not *strictly* proper on C*. It's strictly proper restricted to P, and it's proper on C*, but not strictly so.
ReplyDeleteAnother worry about the scoring rule in the post: is it ‘truth-directed’?
ReplyDeleteFor probabilistically consistent credences, a higher (or lower) c{n} gives higher (or lower) s(c)(n), as it should. But if a small change makes a probabilistic credence non-probabilistic, the score s(c)(n) jumps to 1/2(n+1). Some of these jumps will be in the ‘wrong’ direction.
Yeah, but it's not *clear* that truth-directedness is actually a condition you want for the inconsistent cases, if we think in terms of epistemic utilities.
ReplyDeleteIsn’t truth-directedness, or something like it, essential if a scoring rule is to be used in a dominance argument for probabilism?
ReplyDeleteAn extreme example: the accuracy scoring rule score 1 in all worlds if the credence is probabilistically consistent, score 0 in all worlds if it’s not trivially gives dominance. But using it to justify probabilism would be straight-up question-begging.
For a dominance argument to have force, there must be grounds independent of probabilism to take the scoring rule used as reasonable. (Note that propriety is not independent of probabilism: if you doubt probabilism, why would you use expectation?) Truth-directedness in one such ground.
Of course, there are problems. Kelley “On Accuracy and Dominance with Infinite Opinion Sets” (https://arxiv.org/abs/2007.14490) gives an example (p7) in which dominance fails for the infinite Brier score.
I wrote that it's open what happens when you require additivity. But this is easy to settle. The additive score would have to be defined over a countable number of events only, but any infinite sigma-algebra is uncountable. So let E be some event that isn't included in the sum. Let p be any probability and let p* be a credence that agrees with p on every event other than E. Then p* is not a probability, but it has the same score as a probability, so the score isn't strictly proper. Thus, there is no strictly proper additive score in this infinite case.
ReplyDeleteI just noticed that in the example in the post, s(c) is not even non-strictly dominated by s(p) (where c is a non-probability and p is a probability).
ReplyDeleteI could be wrong, but it's looking to me like there is no continuous strictly proper score in this context, where the space of credences has the product topology on [0,1]^(2^Omega).
ReplyDeleteI'm not sure about what happens if we put the l^infty topology on the space of credences, though.