Pokémon are banned if they become too powerful for their fellow Pokémon to handle, or they become what is called a "click this Pokémon to win". Usually the "click" Pokémon aren't banned if they have certain checks and counters, like Talonflame. While Talonflame is OP, it is still crippled by Rock-types, bulky Waters, and the omnipresent Stealth Rock. Pokémon are more often than not banned due to their overwhelming power and/or utility, like Greninja.
Greninja, while nothing special at first glance, becomes a ban-worthy threat when you look at its bility, which gives STAB to every move it uses, and it's incredible movepool and Speed stat. Greninja was banned because not only did it have a wicked ability, it could also pull off more than one role (Spiker, Fairy-killer, VoltTurn, etc.)
Other Pokémon are banned for being a "click this Pokémon to win". This was the case with Mega Kangaskhan, Mega Salamence, and Mega Rayquaza. Each of these posess blistering speed, rediculous offensive prowess and fantastic abilities. The result was MegaKhan and MegaMence being banned to Ubers, and M-Ray being banished to his own tier.
Another example of a ban-worthy Pokémon is Mega Metagross, currently undergoing a Suspect Test that could ban it to Ubers. This is an excerpt from the Smogon thread concerning him:
>I'd say ban, most likely
M-Metagross has the speed and power to break through most Pokemon in a couple hits and coverage moves to clear out anything that might stop it. Ferrothorn can't handle Hammer Arm (nor can Heatran), Rotom-W gets rekt by Zen Headbutt, and water types are vulnerable to Grass Knot. Fairies are usually just bait for it to switch in, as even Clefable's Flamethrower doesn't damage it as much as it would other steel types.
It has checks, but most of them have trouble switching near full health, and any reasonably built team can handle the threat of Choise Scarf EQ or Shadow Ball from Ground types or Gengar. Once in play and at an advantage, players must either let one of their walls take a hard hit or risk getting crushed by M-Metagross's ridiculous attack stat, meaning there's a good chance that something's gonna die.
M-Meta is also fast enough to require that some worthy opponents are running a scarf, since few Pokemon in speed tiers above it can reliably OHKO it. Overall, it's one of those Pokemon that is so threatening that you need to have multiple ways of dealing with it, since just one weak link can give M-Metagross multiple opportunities to switch in and sweep.
For more on banned, to-be-banned, and suspected Pokémon, go to the Smogon forums. For more on Mega Metagross' Suspect Test, go here.