You forgot to add that people have a tendency not to like people who "aren't like us". This is true of many things, including many stupid things like "not talking like us" or "not wearing the same clothes" etc.

Sexuality is a major part of people's lives, and thus something where one could be "really not like us", hence the hostility to people who practice a different lifestyle.

Not saying this justifies it, it doesn't. But that is probably the main source. Most other excuses are merely used to explain this impulse.