With the exception of the first gym, yes you can completely skip Brawly and beat the other gym leaders first, then come back to get his badge so you can get to the E4. You can also skip Winona in the same way, in case you have trouble with her gym.
Source: I've done it, and here.