Conciliationism holds that in cases of peer disagreement the two peers should move to a credence somewhere between their individual credences. In a recent post I presented a toy model of error of reasoning on which conciliationism was in general false. In this post, I will present another toy model with the same property.
Bayesian evidence is additive when instead of probability p one works with log-odds λ(p)=log(p/(1 − p)). From that point of view, it is natural to model error in the evaluation of the force of evidence as the addition of a normally-distributed term with mean zero to the log-odds.
Suppose now that Alice and Bob evaluate their first-order evidence, which they know they have in common, and come to the individual conclusions that the probability of some Q is α and β respectively. Moreover, both Alice and Bob have the above additive model of their own error-proneness in the evaluation of first-order evidence, and in fact they assign the same standard deviation σ to the normal distribution. Finally, we assume that Alice and Bob know that their errors are independent.
Alice and Bob are good Bayesians. They will next apply a discount for their errors to their first-order estimates. You might think: “No discount needed. After all, the error could just as well be negative as well as positive, and the positive and negative possibilities cancel out, leaving a mean error of zero.” That’s mistaken, because while the normal distribution is symmetric, what we are interested in is not the expected error in the log-odds, which is indeed zero, but the mean error in the probabilities. And once one transforms back from log-odds to probabilities, the normal distribution becomes asymmetric. A couple of weeks back, I worked out some formulas which can be numerically integrated with Derive.
|First-order probability||σ||Second-order probability|
So, for instance, if Alice has a first-order estimate of 0.90 and Bob has a first-order estimate of 0.95, and they both have σ = 1 in their error models, they will discount to 0.87 and 0.93.
Let the discounted credences, after evaluation of the second-order evidence, be α* and β* (the value depends on σ).
Very good. Now, Alice and Bob get together and aggregate their final credences. Let’s suppose they do so completely symmetrically, having all information in common. Here’s what they will do. The correct log-odds for Q, based on the correct evaluation of the evidence, equals Alice’s pre-discount log-odds log(α/(1 − α)) plus an unknown error term with mean zero and standard deviation σ, as well as equalling Bob’s pre-discount log-odds log(α/(1 − α)) plus an unknown error term with mean zero and standard deviation σ.
Now, there is a statistical technique we learn in grade school which takes a number of measurements of an unknown quantity, with the same normally distributed error, and which returns a measurement with a smaller normally distributed error. The technique is known as the arithmetic mean. The standard deviation of the error in the resulting averaged data point is σ/n1/2, where n is the number of samples. So, Alice and Bob apply this technique. They back-calculate α and β from their final individual credences α* and β*, they then calculate the log-odds, average, and go back to probabilities. And then they model the fact that there is still a normally-distributed error term, albeit one with standard deviation σ/21/2, so they adjust for that to get a final credence α** = β**.
So what do we get? Do we get conciliationism, so that their aggregated credence α** = β** is in between their individual credences? Sometimes, of course, we do. But not always.
Observe first what happens if α* = β*. “But then there is no disagreement and nothing to conciliate!” True, but there is still data to aggregate. If α* = β*, then the error discount will be smaller by a factor of the square root of two. In fact, the table above shows what will happen, because (not by coincidence) 0.71 is approximately the reciprocal of the square root of two. Suppose σ = 1. If α* = β* = 0.81, this came from pre-correction values α = β = 0.85. When corrected with the smaller normal error of 0.71, we now get a corrected value α** = β** = 0.83. In other words, aggregating the data from one another, Alice and Bob raise their credence in Q from 0.81 to 0.83.
But all the formulas here are quite continuous. So if α* = 0.8099 and β* = 0.8101, the aggregation will still yield a final credence of approximately 0.83 (I am not bothering with the calculation at this point). So, when conciliating 0.8099 and 0.8101, you get a final credence that is higher than either one. Conciliationism is thus false.
The intuition here is this. When the two credences are reasonably close, the amount by which averaging reduces error overcomes the downward movement in the higher credence.
Of course, there will also be cases where aggregation of data does generate something in between the two data points. I conjecture that on this toy model, as in my previous, this will be the case whenever the two credences are on opposite sides of 1/2.