And one more post on comparing experiments

In my last couple of posts, starting here, I’ve been thinking about comparing the epistemic quality of experiments for a set of questions. I gave a complete geometric characterization for the case where the experiments are binary—each experiment has only two possible outcomes.

Now I want to finally note that there is a literature for the relevant concepts, and it gives a characterization of the comparison of the epistemic quality of experiments, at least in the case of a finite probability space (and in some infinite cases).

Suppose that Ω is our probability space with a finite number of points, and that F_Q is the algebra of subsets of Ω corresponding to the set of questions Q (a question partitions Ω into subsets and asks which partition we live in; the algebra F_Q is generated by all these partitions). Let X be the space of all probability measures on F_Q. This can be identified with an (n−1)-dimensional subset of Euclidean Rⁿ consisting of the points with non-negative coordinates summing to one, where n is the number of atoms in F_Q. An experiment E also corresponds to a partition of Ω—it answers the question where in that partition we live. The experiment has some finite number of possible outcomes A₁, ..., A_m, and in each outcome A_i our Bayesian agent will have a different posterior P_Ai = P(⋅∣A_i). The posteriors are members of X. The experiment defines an atomic measure μ_E on X where μ_E(ν) is the probability that E will generate an outcome whose posterior matches ν on F_Q. Thus:

• μ_E(ν) = P(⋃{A_i:P_Ai|_FQ=ν}).

Given the correspondence between convex functions and proper scoring rules, we can see that experiment E₂ is at least as good as E₁ for Q just in case for every convex function c on X we have:

• ∫_Xcdμ_E2 ≥ ∫_Xcdμ_E1.

There is an accepted name for this relation: μ_E2 convexly dominates μ_E1. Thus, we have it that experiment E₂ is at least as good as experiment E₁ for Q provided that there is a convex domination relation between the distributions the experiments induce on the possible posteriors for the questions in Q. And it turns out that there is a known mathematical characterization of when this happens, and it includes some infinite cases as well.

In fact, the work on this epistemic comparison of experiments turns out to go back to a 1953 paper by Blackwell. The only difference is that Blackwell (following 1950 work by Bohnenblust, Karlin and Sherman) uses non-epistemic utility while my focus is on scoring rules and epistemic utility. But the mathematics is the same, given that non-epistemic decision problems correspond to proper scoring rules and vice versa.