The only true downsides to it are as follows: The Exp. Share can make the game a bit too easy.
If you are trying to do something with a low level Pokemon without adding EV's and Exp. you should turn it off in case you get caught in trainer battle.
I'm sure that there are a few other reasons that various people would turn it off for, but those are the most noticable and it really doesn't matter if you can or can't turn it off, but I suppose I can understand your curiosity.