Forward-looking scoring rules

An accuracy scoring rule assigns a score to a probability function representing an agent’s credences, ostensibly measuring how close that probability function is to the truth. The score s(p) of a probability function p is a random variable, because the value of the score depends on what is actually true, i.e., on where we are in the probability space.

A proper scoring rule (on probabilistic credences) satisfies the propriety inequality

1. E_ps(p) ≥ E_ps(q)

which says that the expected score of your current lights—your current credences p—by your current lights is optimal: you won’t improve your expected score (by your current lights) by switching to a different credence q.

You can think of a proper scoring rule as representing the epistemic utility of having a credence p.

But now let’s think about things dynamically. In the future, you will receive additional evidence. As a good Bayesian agent, you will update on this evidence by conditionalization. Perhaps instead of thinking about maximizing your current score, you should think about maximizing your future score. Maybe your true epistemic utility is the score you will end up with after all the future evidence is in.

A simple model of this is as follows. There is some finite partition I = (I₁,...,I_n) of your probability space Ω with each cell I_i of the partition representing a possibility for what you might learn given future evidence. Your current credence function is p, and p(I_i) > 0 for all i. There is then a random credence function p_I where p_I(ω) is the credence function you will have once the evidene is in if you are at ω ∈ Ω. In other words, p_I(ω)(A) = p(A∣I_i) where I_i is the member of the partition that contains ω. (Technically, the function that maps ω to p_I(ω)(A) is equal to the conditional probability p(A∣G) where G is the algebra generated by I.)

Now, given a proper scoring rule s, define a new scoring rule s_I as follows:

2. s_I(p)(ω) = s(p_I(ω))(ω).

Your s_I-score for p at ω then represents the score you will have at ω once you learn which cell of the partition I you are in.

Theorem: The scoring rule s_I is proper if s is proper.

Note that s_I won’t be strictly proper (i.e., (1) won’t always have strict inequality when p and q are distinct) if I has two or more cells, because p_I and q_I are going to be the same if p and q assign different probabilities to the cells, but have the same conditional probabilities on each cell. But it might still be the case that s_I is strictly proper with respect to some relevant subfield of Ω—that needs some further investigation.

Suppose now you are a Bayesian agent who is guaranteed to consciously live for n moments. In each moment, new information comes in. Thus, we have a sequence J₀, ..., J_n of finer and finer partitions, with J₀ being the trivial partition, and with p_Jk representing the credence you will have at time k. Your overall epistemic lifetime score is then:

3. s_Σ(p) = ∑_ks_Jk(p).

It follows from the Theorem that s_Σ is a proper scoring rule if s is. And if J₀ is the trivial partition, then s_J0 = s, and so if s is strictly proper, then the lifetime score s_Σ is strictly proper, since the sum of a strictly proper rule and a proper rule is strictly proper. So, lifetime scores are strictly proper if they are constructed from an instantaneous score—in the above toy model.

Alas, the toy model is not fully adequate, because it is random when we will die, and so our lifespan doesn’t have a fixed sequence of moments. Once we take into account the randomness of when we will die, the overall epistemic lifetime score might stop being proper: this needs further investigation.

Proof of Theorem: By the Greaves and Wallace Theorem, an optimal method of updating credences with respect to expected proper score is by Bayesian conditionalization. Apply the Greaves and Wallace Theorem to the scoring rule s and the starting credence p with the following two strategies:

A. Bayesian conditionalization on the true cell of I.

B. Switch your credence from p to q, then apply Bayesian conditionalization on the true cell of I.

Saying that (A) is at least as good as (B) is equivalent to the the proper scoring rule inequality (1) for s_I.