Yes contrary only affects the owner of the ability. Eg. If Serperior uses leaf storm, it's Special Attack rises, but if it uses swords dance, it's attack decreases.
On the same example, if the enemy uses swagger on Serperior, Serperior's attack will be reduced (because of contrary.)
BUT if Serperior uses a similar move on the opponent, lowering their stats.... Their stats are still lowered, they are not raised UNLESS the enemy is also a contrary user. In a nut shell, contrary only affects the USER's stat changes, reversing them all.