Based on your tags, you seem to be referring to in-game, so I will ignore simulators... and people who enforce Smogon metagames in forums.
The only place to see the word "BANNED" in the game, that I know of are the battle facilities like the Battle Maison or Battle Institute in X&Y.
It's most likely that you are attempting to use one of the following in those places:
Mythical Pokemon - Pokemon that are intentionally overpowered by the creators. Not simply legends. This includes all box cover legendaries (Lugia, Ho-oh, Kyogre, Groudon, Palkia, Dialga, Reshiram, Zekrom, Xerneas, Yveltal), the third legendary in a trio (Rayquaza, Giratina, Kyurem formes, Zygarde), and Mewtwo.
Event Pokemon - Mew, Celebi, Jirachi, Deoxys, Shaymin, Darkrai, Arceus, Victini, Keldeo, Meloetta, Genesect, Diancie
ONLINE BATTLE SPOT:
In addition to Mythical and Event Pokemon, some special metagames/battles come with various restrictions, like either restricting to Kalos Bred only or Kalos Dex only or both. Or themed battles like the Eevee-only friendly.
You can't trade Event Pokemon, or Pokemon with certain ribbons, on the GTS or Wonder Trade.