On the profile page.
No, if you remove a vote it will not show up on your page. If you downvote, you will lose 2 points and a downvote will be added under "Gave Out". If you remove the downvote by clicking the arrow, you'll lose that downvote from the aforementioned section and gain your points back. If I were to upvote you and then remove it, it will not raise the amount of upvotes I have given out nor take away from point. It won't goes as a downvote or an upvote unless you do the perfromed action and leave it that way.
Hope I helped.