For all previous games it's based on data directly extracted from the game's code. All Pokemon have a certain chance of appearing on each route. We convert that to common/uncommon etc to make it easier.
For newer games like X/Y it will be estimated based on experience playing the game and seeing how many of each Pokemon appear.