Peer disagreement, conciliationism and a toy model

Let suppose that Alice and Bob are interested in the truth of some proposition Q. They both assign a prior probability of 1/2 to Q, and all the first-order evidence regarding Q is shared between them. They evaluate this first-order evidence and come up with respective posteriors α and β for Q in light of the evidence.

Further, Alice and Bob have background information about how their minds work. They each have a random chance of 1/2 of evaluating the evidence exactly correctly and a random chance of 1/2 that a random bias will result in their evaluation being completely unrelated to the evidence. In the case of that random bias, their output evaluation is random, uniformly distributed over the interval between 0 and 1. Moreover, Alice and Bob’s errors are independent of what the other person thinks. Finally, Alice and Bob’s priors as to what the correct evaluation of the evidence will show is uniformly distributed between 0 and 1.

Given that each now has this further background information about their error-proneness, Alice and Bob readjust their posteriors for Q. Alice reasons thus: the probability that my first-order evaluation of α was due to the random bias is 1/2. If I knew that the random bias happened, my credence in Q would be 1/2; if I knew that the random bias did not happen, my credence in Q would be α. Not knowing either way, my credence in Q should be:

1. α^* = (1/2)(1/2)+(1/2)α = (1/2)(1/2 + α).

Similarly, Bob reasons that his credence in Q should be:

2. β^* = (1/2)(1/2 + β).

In other words, upon evaluating the higher-order evidence, both of them shift their credences closer to 1/2, unless they were at 1/2.

Next, Alice and Bob pool their data. Here I will assume an equal weight view of how the data pooling works. There are now two possibilities.

First, suppose Alice and Bob notice that their credences in Q are the same, i.e., α^* = β^*. They know this happens just in case α = β by (1) and (2). Then they do a little Bayesian calculation: there is a 1/4 prior that neither was biased, in which case the equality of credences is certain; there is a 3/4 prior that at least one was biased, in which case the credences would almost certainly be unequal (the probability that they’d both get the same erroneous result is zero given the uniform distribution of errors); so, the posterior that they are both correct is 1 (or 1 minus an infinitesimal). In that case, they will adjust their credences back to α and β (which are equal). This is the case of peer agreement.

Notice that peer agreement results in an adjustment of credence away from 1/2 (i.e., α^* is closer to 1/2 than α is, unless of course α = 1/2).

Second, suppose Alice and Bob notice that their credences in Q are different, i.e., α^* ≠ β^*. By (1) and (2), it follows that their first-order evaluations α and β were also different from one another. Now they reason as follows. Before they learned that their evaluations were different, there were four possibilities:

• EE: Alice erred and Bob erred
• EN: Alice erred but Bob did not err
• NE: Alice did not err but Bob erred
• NN: no error by either.

Each of these had equal probability 1/4. Upon learning that their evaluations were different, the last option was ruled out. Moreover, given the various uniform distribution assumptions, the exact values of the errors do not affect the probabilities of which possibility was the case. Thus, the EE, EN and NE options remain equally likely, but now have probability 1/3. If they knew they were in EE, then their credence should be 1/2—they have received no data. If they knew they were in EN, their credence should be β, since Bob’s evaluation of the evidence would be correct. If they knew they were in NE, their credence should be α, since Alice’s evaluation would be correct. But they don’t know which is the case, and the three cases are equally likely, so their new credence is:

3. α^** = β^** = (1/3)(1/2 + α + β) = (1/3)(2α^* + 2β^* − 1/2).

(They can calculate α and β from α^* and β^*, respectively.)

Now here’s the first interesting thing. In this model, the “split the difference” account of peer disagreement is provably wrong. Splitting the difference between α^* and β^* would result in (1/2)(α^* + β^*). It is easy to see that the only case where (3) generates the same answer as splitting the difference is when α^* + β^* = 1, i.e., when the credences of Alice and Bob prior to aggregation were equidistant from 1/2, in which case (3) says that they should go to 1/2.

And here is a second interesting thing. Suppose that α^* < β^*. Standard conciliationist accounts of peer disagreement (of which “split the difference” is an example) say that Alice should raise her credence and Bob should lower his. Does that follow from (3)? The answer is: sometimes. Here are some cases:

• α^* = 0.40, β^* = 0.55, α^** = β^** = 0.47
• α^* = 0.55, β^* = 0.65, α^** = β^** = 0.63
• α^* = 0.60, β^* = 0.65, α^** = β^** = 0.67
• α^* = 0.60, β^* = 0.70, α^** = β^** = 0.70.

Thus just by plugging some numbers in, we can find some conciliationist cases where Alice and Bob should meet in between, but we can also find a case (0.60 and 0.70) where Bob should stand pat, and a case (0.60 and 0.65) where both should raise their credence.

When playing with numbers, remember that by (1) and (2), the possible range for α^* and β^* is between 1/4 and 3/4 (since the possible range for α and β is from 0 to 1).

What can we prove? Well, let's first consider the case where α^* < 1/2 < β^*. Then it's easy to check that Bob needs to lower his credence and Alice needs to raise hers. That's a conciliationist result.

But what if both credences are on the same side of 1/2? Let say 1/2 < α^* < β^*. Then it turns out that:

4. Alice will always raise her credence

5. Bob will lower his credence if and only if β^* > 2α^* − 1/2

6. Bob will raise his credence if and only if β^* < 2α^* − 1/2.

In other words, Bob will lower his credence if his credence is far enough away from Alice’s. But if it’s moderately close to Alice’s, both Alice and Bob will raise their credences.

While the model I am working with is very artificial, this last result is pretty intuitive: if both of them have credences that are fairly close to each other, this supports the idea that at least one of them is right, which in turn undoes some of the effect of the α → α^* and β → β^* transformations in light of their data on their own unreliability.

So what do we learn about peer disagreement from this model? What we learn is that things are pretty complicated, too complicated to encompass in a simple non-mathematical formulation. Splitting the difference is definitely not the way to go in general. Neither is any conciliationism that makes the two credences move towards their mutual mean.

Of course, all this is under some implausible uniform distribution and independence assumptions, and a pretty nasty unreliability assumption that half the time we evaluate evidence biasedly. I have pretty strong intuitions that a lot of what I said depends on these assumptions. For instance, suppose that the random bias results in a uniform distribution of posterior on the interval 0 to 1, but one’s prior probability distribution for one’s evaluation of the evidence is not uniform but drops off near 0 and 1 (one doesn’t think it likely that the evidence will establish or abolish Q with certainty). Then if α (say) is close to 0 or 1, that’s evidence for bias, and a more complicated adjustment will be needed than that given by (1).

So things are even more complicated.