Why, the Kelpsy Berry is your solution!
The Kelpsy Berry is a berry that, in Generation V and VI, reduces the Attack EV back to 100 (assuming it's over 100) and then decreases the Attack EV by 10.
You can also use the Reset Bag in Super Training, but that removes all EVs gained in the Attack EV entirely.
Hope I helped. :)