I think it would be a good idea. We removed giving points for up - voting for the same reason, if you ask a question that doesn't get down voted, that's a free 4 points. And users who want quick points can do that, and since we have a point limit for voting, anyone who can ask 20 questions that don't get down-voted, can vote on things.
Which is a bit scary, if you ask me. Just a thought.