You just lose speed when in battle, like you are holding an Iron Ball. It is not permanent. Speed goes back to normal when you take it off. Same goes for all the other Power Items.
It would be very counterproductive if it was permanent.
Source: Experience EV training in Speed.