i just downvoted an answer, and found that my points were lower. i had same number of downvotes and NO flags, so i thought about this...
Yes, you lose 2 points for down voting things. You do not lose points for flagging. It is like this to prevent spam-downvoting.