The Power Band only lessens the speed in battle, so you might find yourself suddenly outspeeding Pokemon that you couldn't when you had the Band on. The speed will return after the Band has been taken off the Pokemon.
Overall: The Power Band will lessen your speed while holding it only.
Source: Me and My Broken DS...