Case 1: A child is drowning in a dirty pond. You can easily pull out the child. But you’ve got cuts all over your dominant arm and the water is full of nasty bacteria and medical help is a week away. If you go in the water to pull out the child, your arm will get infected, become gangrenous and in a week it will be amputated. There will be no social losses or gains to you.
Case 2: A child is drowning in a clean pond. You can easily pull out the child. But the child is a member of a despised minority group, and you will be ostracized by your friends and family for life for your rescue. There will be no physical losses or gains to you.
Here is my intuition. In both cases, it would be a good thing to rescue the child. But in Case 1, unless you have special duties (e.g., it’s your own child), you do not have a duty to rescue given the physical costs. In Case 2, however, you do have a duty to rescue, despite the social costs.
The difference between the two cases does not, I think, lie in its being worse to lose an arm than to be ostracized. Imagine your community has a rite of passage that involves swimming in the dirty pond with the cuts on your arm, and you’d be ostracized if you don’t. You might well reasonably judge it worthwhile—but still, I think, the intuition remains that in Case 2 you ought to pull out the child, while in Case 1 it’s supererogatory. So it seems then you might have a duty to undertake the greater sacrifice (facing social stigma in Case 2) without a duty to undertake the lesser sacrifice (amputation in Case 1). But for simplicity let’s just suppose that the harms to you in the two cases are on par.
Is it that physical harm excuses one from the duty to rescue the child but social harm does not? I don’t think so.
Case 3: A child is being murdered by drowning in a clean pond. You can easily pull out the child. But if you do, the murderer will punish you for it by transporting you away from your home community to a foreign community where you will never learn the difficult language and hence will not have friends.
We can set this up so the harm in all three cases is equal. But my intuition is that Case 2 is like Case 1: in both cases it is supererogatory to rescue the child but there is no duty.
In Cases 2 and 3 we have equal social harms, but I feel a difference. (Maybe you don’t!) Here’s one consideration that would explain the difference. That an action gains one the praise and friendship of bigots qua bigots does not count much in favor of the action, even if, and perhaps even especially if, such praise and friendship would make one’s life significantly more pleasant. Similarly, that an action loses one the friendship of bigots, and does so precisely through their bigotry, is not much of a consideration against the action. I say “not much”, because there might be instrumental gains and losses in both cases to be accounted for.
Here’s a second consideration. Perhaps if I refrain from doing something directly because doing it will lose me bigots’ friendship or gain me their stigma, I am thereby complicit in the bigotry. In Case 2, then, I need to ignore the direct loss of goods of social connectedness in considering whether to rescue the child. I need to say to myself: “When those are the conditions of their friendship, so much the worse for their friendship.” In Case 3, I have similar social losses, but I don't lose the friendship of bigots qua bigots, so the loss counts a lot more.
But note that one can still legitimately consider the instrumental harms from the loss of goods of social connectedness. Consider:
Case 4: A child is drowning in a clean pond, but you have a wound that will become gangrenous and force amputation absent medical help. You can easily pull out the child. But the child is a member of a despised minority group, and if you rescue the child, the only doctor in town will refuse to have anything to do with you. As a result, your wound will become gangrenous by the time you find another doctor, and you will require amputation.
I think in Case 4, you are permitted not to rescue the child, just as in Case 1.