The rules for catching Pokémon in a Nuzlocke are:
• You must catch the first Pokémon you see on a route.
• That is the only Pokémon you can catch on that route.
• If you encounter your first Pokémon in a Double Battle, you get to choose which one you want.
Those are the basic rules, taken from here.
The great thing about Nuzlocke is that, aside from the fundamental rules, you get to invent your own as you play. For example, in my Nuzlocke, I can fail to capture the first Pokémon I see, then catch another. But I have to catch the same Pokémon. For example, if I encounter a Buizel that KOs itself from confusion, I can keep trying until I catch a Buizel. But only a Buizel.
The best part of a Nuzlocke is the freedom of choice; as long as the core rules are intact, you can do whatever you want. Have fun!
