The morality of fantasy I think there is something to say about a person feeling weird about some fantasies afterwards, so to make it something to avoid, but one has to wonder if some fantasies are not just an expression of something that is prior to morality. One can be turned on, for example, by many things in a rape fantasy; to hurt, dominate, control, be feared, and whatever. I don't think these are not inhumane things to want to let out in fantasies, even should it involve something quite horrible, as long as one is strong enough to completely lock their expression in this way out of the everyday life (minus the fantasies) and indeed more strongly push them away the stronger the urges are. This fantasy then, as I see it, is just an expression of who-knows-what, with no connection to reality.
Also, as has been mentioned, there might be some people that can't lock them out and fantasies just strengthen the urges, but this doesn't bring them to the moral realm without an actual action. If one actually carries out a disturbing fantasy, then we can say that she shouldn't have fantasied about it, but chances are, she was already on the way to the action due to her weakness before the urge. If one however, say a paedophile, never carries out an action, then she has not done anything wrong, and should maybe even be commended for it. Since the fantasy has not been acted out however, the reality of the urge is, I think, to be questioned. It never found its steps into reality, so maybe what the urge was about was not what is thought to be its end.
And indeed, is any willing of something completely known? We don't know their causes, as is shown by the constant belief in "free will". So any actual decision is based on something irrational, and this irrational I think finds expression in fantasies. There's a lot of taking away in them; they don't necessarily consider people as people with a past, the actual thinker can appear as someone completely different, the imagined visual image is unclear and emphasises again and again that which captures the thinkers attention right now, depending on what his drive right now is. You can, for example, be quite certain that the very unrealistic, idealised portrayal of children in Japanese hentai has quite a few fans that would never actually like to see such things in photos, and so on. Would imagining the drawings be that different from imagining an actual child? I don't think the visible difference would be enormous, nor the difference between other content.
I feel like I'm all over the place, but the point is this: Societal life can be incredibly restrictive, and many aspects of the human nature can't be expressed in it, but are pushed down by force as something bad. To find ways to let these urges out without harm is maybe even a necessity for the society. Moreover, even bypassing into the "wrong" might work as an encouragement for disturbing fantasies, so the real end of the urge from which the fantasy is born is even more unclear. The thing to be judged are the actions and the quality of the person (unfortunately also shown through actions).