Why do people bother to be actively or aggressively phobic?

I would never be so rude as to tell someone I had a problem with their lifestyle, mostly because it's nothing to do with me. As long as I'm not hurting anyone, I pay my own way and I'm legal, moral and ethical, why on earth would anyone care what I do in bed? Are they that bored?