Sometimes we fantasize about specific others behaving a certain way with regard to us. Sexual fantasies are one species of the genus I am interested in, but the genus is wider than that. There can, for instance, be fantasies about the recognition of our excellences, about others doing something humiliating, about climbing Mount Everest with one's best friend, etc.
To fantasize about a situation is more than just to think about the possibility of it. As in the case of hoping, there is a positive attitude towards the situation, though unlike in the case of hope, there need be no expectation. The positive attitude by itself shows that fantasizing about a bad situation (e.g., about one's being cruel to someone or one's engaging in an illicit flirtation) is wrong—for, surely, our attitudes should be appropriate to their object, and the attitude towards something bad should negative. In such a case, we have what Aquinas calls the sin of "morose delectation". Moreover, beyond a positive attitude, there is a first-person involvement in a fantasy--one reacts emotionally to it in somewhat the way one would were it real.
What I said so far shouldn't be controversial, though in the past I've had trouble getting some students to accept that morality governs the life of the mind.
But I want to note a different kind of badness in fantasies involving the behavior of specific others, even when the situation fantasized about is not actually a bad one. This kind of badness occurs when the fantasy does not respect others as autonomous persons. The fantasizer is, after all, in charge of the situation. She is like film director, telling this actor to do this that actor to say that. But unlike a real film director who does this in cooperation with actors who have read the script and agreed to act according to it, the typical fantasizer is arranging the persons in her mind without any cooperation, all on her own. And herein lies both the attraction and the danger of the fantasy. The fantasizer in creating the fantasy is in a sense more powerful than God in creating the world. For while God cannot make a person freely do something (this is true by the relevant definition of "freely"), the fantasizer can. It can, for instance, be a part of the fantasy that some persons freely fawn on her. This attitude of being in charge of others can be a way of using them.
I want to qualify this a little. There is a certain respecting of autonomy if the behavior of the fantasy's characters is constrained by the real-life behavior or commitments of the persons. If I have had a number of delightful conversations with George, there perhaps is nothing wrong in fantasizing about another, since in doing so I am constrained by George's actual character, and thus he is to some extent autonomous even as found in my mind. If I do this well, I might even find myself rebuked by fantasy-George in the course of the fantasized conversation. Likewise, if someone has undertaken a morally licit commitment to do something with me, it does not seem problematic to look forward vividly to that activity. Again, the actual person has had a moment of autonomy in the creation of the fantasy.
But the more the fantasizer is in charge in arranging the behavior of the characters in the fantasy for her own gratification, the more problematic this fantasizing is, as it is a failure to respect the fact that others are independent persons, not subordinate to her pleasure.
On obvious objection is that I am confusing fiction and reality. The student who fantasizes about me saying that his shoddy paper, which he whipped off during the half hour before it was due, was the best I have ever read is not actually making me do anything. This is true, I respond. But he is in an important way using me for his own gratification. His fantasy gets its life from my reality. That I am not actually physically affected by the fantasy does not mean that I am not used—certainly, the voyeur's victim is used by the voyeur even if the victim does not find out.
It is true that when sane people fantasize, they can typically distinguish fact from fiction. But at the same time, what gives pleasure in the fantasy is a deliberate mental relaxing of the distinction, a willing suspension of disbelief. To treat the characters that inhabit one's fantasy as pawns to be moved in accordance with one's desires for one's gratification is seriously problematic, and it develops a disrespectful habit of the mental treatment of others. Even if one is right that this habit will not overflow into controlling behavior—and how can one be sure of that?—the mental attitudes are themselves morally bad.