Like if someone has upvoted then let's say that user got down-voted (or any other way to lose points) and loses their privileges to upvote, would the upvotes they gave out beforehand be removed?
They would not.