The worst thing someone actually ever said to my face was only "I'll be blunt, I don't like you" in a bar I used to work in, but that sent four of my other workmates into rapturous laughter because everyone else who worked there hated her guts for being a stuck up cow.
I have a few others that stick in my mind, mostly people accusing me subtly (or not so subtly) of being a bit strange.
Me : "Am I weird?"
Friend (lengthy pause) : "Well.... you're not normal."
Friend 1 : "(Stiffy)!!!"
Friend 2 : "Christ, what's he done now??"
Girl (to friend, about me) : "He's got an incredible mind, hasn't he?"
Girl (to friend, about both of us) : "You're so immature."
Friend : "I'm immature. HE shouldn't be allowed out. Ever."
I also had a female friend tell me "You're an animal" after watching me lick the sugar from the rim of the glass my margarita was served in. This being a female friend who has told me that if her long-term, committed relationship ever broke down, I'd be the first guy she'd want to fuck. This thought keeps me awake at night sometimes.