Personally, it annoys me when people say that about me. It's not as if I've gone out of my way to make them uncomfortable. I do think they mean it in a nice way, though I can't figure out how, but it still bothers me.