I've completely rewritten this answer to list the exact points you get for each thing, since it's changed many times since this question was asked. Here we go.
- All users start with 20 points. Think of it like a sign-up bonus ;)
- If your question or answer is up-voted, you gain 10 points.
- If your question or answer is down-voted, you lose 10 points.
- Having your answer selected as the best gives you 20 points.
- Selecting an answer as the best on your question gives you 2 points.
- Down-voting a post loses you 2 points (this is to discourage people downvoting for no reason).
If a post is hidden, points gained from upvotes are lost, but points lost from downvotes stay, they do not come back. However, the -2 from downvoting gets returned eventually I think.
See also: List of privileges and required points