I don't think this would cause fall outs Sciz, because what I believe he means in the description is that, only the user can see what they've up and down voted, no one else can.
However (and I don't mean to be rude here), but the whole idea is kind of pointless, I mean why would you want to take away an up vote that you've given someone? You must of obviously thought the post was good at some point in the past to up vote it, so why take it away? The only reason I can think of is if you had a falling out with someone or something that involves points.
It's pretty obvious to me why you want PM to implement this suggestion, it's so you can see who you've down voted so you can un-down vote them and gain back your points. Well again this is also kind of pointless because it's only -2 points and if you didn't want to lose any points you shouldn't of really down voted anyone in the first place :l
Not to mention when a post is down voted it is sometimes hidden, (if it's wrong or receiving excessive down votes etc.) So since it's hidden it won't come up on the list of posts you've down voted and even if it did, you couldn't un-down vote it. You also can't remove down votes from answers that have been converted into comments, since they're not answers anymore.
