CRYSTAL: No difference. People just prefer Kris because she's the first playable female character and iconic.
RS: No difference in male vs female. Source
EMERALD: The male rival says less text than the female rival, and thus takes up less time. Also, for the Rival 2 fight, the male rival leads Slugma, and the female rival leads Wingull. In that fight you have to set up 2 X Attacks and an X Speed to take down the Grovyle, however the wingull knows growl. As you can only afford a maximum of 3 X Attacks up to this point (and normally only have 2), getting growl from the wingull can be a run killer. The slugma however only knows yawn and ember, and there is a full heal hidden just before Rival 2 as a safety against getting burned by ember. However, because the slugma knows yawn, and male Rival 3 has a Pelliper where female Rival 3 has a wingull, these fights are actually slower. It is still worth it regardless to choose the female though. Source, thanks SS
FRLG: gender doesn't matter. Source
HGSS: Lyra talks too much to Ethan. If you play as Lyra, you don't have to listen to her. Source
DPPt: The girl is slightly faster than the boy for speedrunning because Lucas has less to say than Dawn.
commented Mar 10 by J™
BW/2: Nothing noted here in terms of gender, just game versions.
XY: X is faster because you can keep the Battle Style on Set for the final Lyandre battle, but in Pokémon Y, this is not possible because Lysandre sends his Pokémon out in a different order. The time difference is quite small in comparison to version differences of other paired games.
When you pick the girl, your rival (Calem) will have a slower Meowstic at Coumarine City, meaning that your Hawlucha is more likely to outspeed it, which makes the run safer. Source
TL;DR: it's usually faster in either dialogue or battles
