There is actually a limit, although it's quite high currently: 40 votes per hour, which includes upvotes and downvotes.
It's hard to find a balance though. For example if we said the limit should be 100 per day, that works out at only 4 per hour which is too low as most people would come on for less than an hour a day.
As PX said, the moderators can keep track of downvotes. Also since you lose 2 points for downvoting, it prevents people doing it too much (especially newer users because then they'd lose the privilege to downvote).