AI and emotion

In some of my work, I use the example of a pill which gives one that warm glow that one has when one has done something sefless and morally good, but which pill one can take when one hasn’t done anything like that, just to feel good about oneself. This is wrong, because that warm glow emotion is too important morally for it to be the subject of counterfeiting. Moreover, I think it remains wrong to take the warm glow pill even if one fully knows that one hasn’t done the morally good deed that it fakes the feeling of.

Generalizing, I think we shouldn’t deliberately induce emotions in contexts where they are inapt when these emotions have a significant amount of moral importance. We shouldn’t induce them in ourselves nor in others. For instance, we shouldn’t try to make others feel like we are their friends when we are not—even if they fully know that that the feeling is misleading.

Now, there is a multitude of significantly morally important interpersonal emotions that are only apt as reactions to another person’s actions. These include feelings of being the object of good- or ill-will, feelings of gratitude or resentment, a feeling of not being alone, and of course a feeling of being a friend. Such emotions have a significant amount of moral importance. We should thus not try to induce them deliberately.

But I think a plausible case can be that current AI chatbots are tuned (both through feedback from users and the system prompt) to produce emotional reactions that are of this interpersonal sort—the communications of the chatbot are tuned to make one feel that one’s concerns are care about. And since the chatbots aren’t persons, the emotions are inapt. The tuning is thus morally wrong, even if any sensible user knows that the chatbot has no cares.

One can, sometimes, have a double-effect justification of inducing misleading emotions, when doing so is an unintended side-effect. However, given that leaked system prompts do in fact have instructions about emotional cadence, it is very implausible to think that the induction of inapt emotions is an unintended side-effect.

A couple of days ago, Anthropic offered me a decent chunk of money for doing some part-time review of the reasoning capabilities of one or more of their models. I turned it down because of moral concerns along the above lines.

I note that double-effect can, however, justify using a chatbot when one does not intend an inapt emotion that one expects in oneself (e.g., I find myself feeling grateful when I get a good AI answer), when the goods gained from the use are sufficient in comparison to the significance of the inapt emotion. But I think the risk should be taken into account.

This is all rather similar to St. Augustine’s infamous concerns about stage drama. But I think one can make a distinction between the cases. Interpersonal emotions can be categorical or hypothetical. Categorical disapproval is apt only when a person has done something morally wrong. But we also have hypothetical disapproval: we can imagine someone hypothetically acting in some situation, and then have a feeling of disapproval towards that hypothetical action. I think there is a real felt difference between these two feelings, just as there is a real felt difference between seeing a sunset and imagining a sunset. And, perhaps, the audience of a dramatic performance one only has—or at least should only have—the more hypothetical feeling.

Lying, acting and trust

A spy's message to his handler about troop movements is intercepted. The message is then changed to carry the false information that the infantry will be on the move without artillery support and sent onward. Did those who changed the message lie?

To lie, one must assert. But suppose the handler finds out about the change. Could she correctly say: "The counterintelligence operatives asserted to us that the infantry would be on the move without artillery support?" That just seems wrong. In fact, it seems similar to the oddity of attributing to an actor the speech of a character (though with the important difference that the actor does not typically speak to deceive). The point is easiest to see, perhaps, where there are first person pronouns. If part of the message says: "I will be at the old barn at 9 pm", it is surely false that the counterintelligence staff asserted they will be at the old barn (even though, quite possibly, they will--in order to capture the handler), but it also doesn't seem right to say that the counterintelligence staff asserted that the spy will be there.

The trust account of lying, defended by Jorge Garcia and others, seems to fit well with this judgment. On this account, to lie is to solicit trust while betraying it. But one can only betray a trust in oneself. The counterintelligence operatives, however, did not solicit the handler's trust in themselves: rather, they were relying on the handler's trust in the spy, and that trust the operatives cannot betray.

But there are some difficult edge cases. What if a counterintelligence operative dons a mask that makes him look just like the spy, and speaks falsehoods with a voice imitating the spy? But what if a spy goes to a foreign country with an entirely fictional identity? I am inclined to think that on the trust account the two cases are different. When one imitates the spy, one relies on the faith and credit that the spy has, and one isn't soliciting trust for oneself. When one dresses up as someone who doesn't exist, I think one is trying to gain faith and credit for oneself, and it seems one is lying. But I am not sure where the line is to be drawn.