Wednesday, July 26, 2023

Committee credences

Suppose the members of a committee individually assign credences or probabilities to a bunch of propositions—maybe propositions about climate change or about whether a particular individual is guilty or innocent of some alleged crimes. What should we take to be “the committee’s credences” on the matter?

Here is one way to think about this. There is a scoring rule s that measures the closeness of a probability assignment to the truth that is appropriate to apply in the epistemic matter at hand. The scoring rule is strictly proper (i.e., such that an individual by their own lights is always prohibited from switching probabilities without evidence). The committee can then be imagined to go through all the infinitely many possible probability assignments q, and for each one, member i calculates the expected value Epis(q) of the score of q by the lights of the member’s own probability assignment pi.

We now need a voting procedure between the assignments q. Here is one suggestion: calculate a “committee score estimate” for q in the most straightforward way possible—namely, by adding the individuals’ expected scores, and choose an assignment that maximizes the committee score estimate.

It’s easy to prove that given that the common scoring rule is strictly proper, the probability assignment that wins out in this procedure is precisely the average  = (p1+...+pn)/n of the individuals’ probability assignments. So it is natural to think of “the committee’s credence” as the average of the members’ credences, if the above notional procedure is natural, which it seems to be.

But is the above notional voting procedure the right one? I don’t really know. But here are some thoughts.

First, there is a limitation in the above setup: we assumed that each committee member had the same strictly proper scoring rule. But in practice, people don’t. People differ with regard to how important they regard getting different propositions right. I think there is a way of arguing that this doesn’t matter, however. There is a natural “committee scoring rule”: it is just the sum of the individual scoring rules. And then we ask each member i when acting as a committee member to use the committee scoring rule in their voting. Thus, each member calculates the expected committee score of q, still by their own epistemic lights, and these are added, and we maximize, and once again the average will be optimal. (This uses the fact that a sum of strictly proper scoring rules is strictly proper.)

Second, there is another way to arrive at the credence-averaging procedure. Presumably most of the reason why we care about a committee’s credence assignments is practical rather than purely theoretical. In cases where consequentialism works, we can model this by supposing a joint committee utility assignment (which might be the sum of individual utility assignments, or might be consensus utility assignment), and we can imagine the committee to be choosing between wagers so as to maximize the agreed-on committee utility function. It seems natural to imagine doing this as follows. The committee expectations or previsions for different wagers are obtained by summing individual expectations—with the individuals using the agreed-on committee utility function, albeit with their own individual credences to calculate the expectations. And then the committee chooses a wager that maximizes its prevision.

But now it’s easy to see that the above procedure yields exactly the same result as the committee maximizing committee utility calculated with respect to the average of the individuals’ credence assignments.

So there is a rather nice coherence between the committee credences generated by our epistemic “accuracy-first”
procedure and what one gets in a pragmatic approach.

But still all this depends on the plausible, but unjustified, assumption that addition is the right way to go, whether for epistemic or pragmatic utility expectations. But given this assumption, it really does seem like the committee’s credences are reasonably taken to be the average of the members’ credences.


Lucas Marques said...

You're a genius, Pruss - it's even a trivia to write this. Something makes me compare him to Leibniz, my philosophy hero. From Brazil!

Attila said...

Wouldn't a median unbiased estimater for the population credence be more appropiate as those are more robust for or with outliers than a mean unbiased estimator for the population credence?!?

I guess, that the following estimator would and could be more robust robust than the simple arithmetic mean:
θ:=cr[(n+1)/2] for the sample size n being odd and
θ:=(cr[n/2]+cr[n/2+1])/2 for n being even.

I guess, that this would and could be more appropiate in some cases.

Alexander R Pruss said...

I think a median doesn't reflect "committee credence" well. Suppose there are three people, with credences 0.1, 0.5 and 0.6. Suppose additional evidence comes in that convinces the two outlier people, but not the 0.5 person, so that they now go to 0.4, 0.5, and 0.9; instead, the 0.5 person reflects on something and goes down 0.49. Intuitively, the committee is more favorable to the proposition now: the majority of the committee has significantly raised their credences, and the remainder has slightly lowered it. Yet the median has gone down to 0.49.

The outliers are also a part of the committee, and their opinions should also affect the committee credence.

I know we have a tendency to think that something likely wrong with outlier values. But that need not be true in the epistemic case. Just as some people have a tendency to excesses of certainty, some people have a tendency to excesses of suspension of judgment.