There is a bit of a quirk in the the user lists where the final page of nagivation is stuck on #334, and it's impossible to view users past that point. So the older users were pushed to a hypothetical 335th page (and so on), which doesn't exist and that's why you can't see them. I tested, and modifying the URL manually doesn't change this.
Source
If there were more pages, downvoted users would be shown. If there were less users, downvoted users would be shown. As neither of these are true, downvoted users are not shown.
Hope I helped!
