Fishing chains are when you constantly fish with one of the Rod items. The more fish you get in a row, the higher your chance of finding a Shiny Pokemon becomes. The following things will break your chain:
- Moving from your position. Once you start chain fishing do not move anywhere or your chain will be broken.
- Pulling in the rod too early or too late.
- Not getting a Pokemon on your rod. This happens naturally and is almost inevitable. However, if you lead your party with a Pokemon with the ability Suction Cups, you are guaranteed to get a Pokemon on your Rod, eliminating this from the list of things that could break your chain.
What you do in battles does not effect your chain; whether you catch the Pokemon, defeat it or run from it, the chain continues.
I also find that accessing a menu in the overworld breaks your chain, though this may be coincidental.
More on chain fishing