If you mean online competitions, like the battle spot, then out of the six Pokemon you have in your team, you can take:
>3 to battle in Singles
>4 to battle in Doubles
>All 6 to battle in Triples
>All 6 to battle in Rotation
>3 to battle in Inverse (special)
If you mean the Battle Maison the single battle challenges you require three (no more, no less) Pokemon.
If you meant Wi-Fi battles against a friend, you have either Flat Rules (which take the number of Pokemon in account in the same way battle spot does), or you can go with normal rules or free battles in which you can take all 6 Pokemon to battle with you.
Source - Experience
Hope I helped!