The more you upset it, then less Affection you'll have for it, and you start losing the bonuses (boosted EXP, higher critical hit ratio, etc.). The less your PKMN is interacted with anyway, its Affection goes down. And according to this question, it may start to completely ignore you in-battle.
Source: experience.
