DT is right, you lose 2 points. The point is to discourage people down voting everything they see for no good reason.
Remember, the person you down vote loses more points (10) for posting something bad. And if it's just retarded spam then a mod will likely hide it fairly quickly. You may get the points back once the posts get fully deleted.