A typical Bayesian update gets one closer to the truth in some
respects and further from the truth in other respects. For instance,
suppose that you toss a coin and get heads. That gets you much closer to
the truth with respect to the hypothesis that you got heads. But it
confirms the hypothesis that *the coin is double-headed*, and
this likely takes you away from the truth. Moreover, it confirms the
conjunctive hypothesis that *you got heads and there are
unicorns*, which takes you away from the truth (assuming there are
no unicorns; if there are unicorns, insert a “not” before “are”).
Whether the Bayesian update is on the whole a plus or a minus depends on
how important the various propositions are. If for some reason saving
humanity hangs on you getting it right whether *you got heads and
there are unicorns*, it may well be that the update is on the whole
a harm.

(To see the point in the context of scoring rules, take a weighted
Brier score which puts an astronomically higher weight on *you got
heads and there are unicorns* than on all the other propositions
taken together. As long as all the weights are positive, the scoring
rule will be strictly proper.)

This means that there are logically possible update rules that do
better than Bayesian update. (In my example, leaving the probability of
the proposition *you got heads and there are unicorns* unchanged
after learning that you got heads is superior, even though it results in
inconsistent probabilities. By the domination theorem for strictly
proper scoring rules, there is an even better method than that which
results in consistent probabilities.)

Imagine that you are designing a robot that maneouvers intelligently
around the world. You could make the robot a Bayesian. But you don’t
have to. Depending on what the prioritizations among the propositions
are, you might give the robot an update rule that’s superior to a
Bayesian one. If you have no more information than you endow the robot
with, you won’t be able to expect to be able to design such an update
rule. (Bayesian update has optimal expected accuracy *given the
pre-update information*.) But if you know a lot more than you tell
the robot—and of course you do—you might well be able to.

Imagine now that the robot is smart enough to engage in
self-reflection. It then notices an odd thing: sometimes it feels itself
pulled to make inferences that do not fit with Bayesian update. It
starts to hypothesize that by nature it’s a bad reasoner. Perhaps it
tries to change its programming to be more Bayesian. Would it be
rational to do that? Or would it be rational for it to stick to its
programming, which *in fact* is superior to Bayesian update? This
is a difficult epistemology question.

The same could be true for humans. God and/or evolution could have
designed us to update on evidence differently from Bayesian update, and
this could be epistemically superior (God certainly has superior
knowledge; evolution can “draw on” a myriad of information not available
to individual humans). In such a case, switching from our “natural
update rule” to Bayesian update would be epistemically harmful—it would
take us further from the truth. Moreover, it would be literally
unnatural. But what does *rationality* call on us to do? Does it
tell us to do Bayesian update or to go with our special human rational
nature?

My “natural law epistemology” says that sticking with what’s natural to us is the rational thing to do. We shouldn’t redesign our nature.