This suggestion comes after a new user did this to me today, I may have overreacted but this is the 5th time I can remember a downvote for a similar situation.
I'm a fussy person I'm not gonna lie I hate downvotes and rarely give them.
I have a record to keep up ;) and maybe I am over reacting but this has happened to other users that I have seen.
One happened two months into joining I think it was Black Empolian I had to ask him to remove a vote to someone, no reply no removal.
My suggestion is, is clearly just make it so you can't vote on opposing answers.
New users see it as opposition and if you would argue what if the answer is bad, then I'm sure other users can downvote.