Most likely because physical attackers use moves that make contact. When you're burnt and you touch something, it hurts. Rough skin hurts your own skin, foul play uses the other Pokemon's physical ability. Special attack is usually a mental thing (i.e. psychic moving things using the power of thought) and so it's harder to hurt your brain by making contact. I see how it could be confusing though. Maybe poison will act as burn for special attack in the future or some new status will come up.