Yes, you can - nothing stops you from going to Mahogany Town. You can still participate in the Team Rocket + Lake of Rage storyline, and get your Red Gyarados, which may be of some use to you in the other two gyms.
To challenge the league, you'll need to go through the two gyms anyways. Also, you won't have Fly, which requires the defeat of Chuck, meaning you'll have to actually walk to get from city to city, which can be quite annoying. If anything do Chuck's gym then skip Olivine so you have Fly at least.
So yeah, it's possible, just extremely inconvenient since you won't have Fly.