>Raises Attack if the foe is of the same gender.
- Bulbapedia
So, your allies will not affect Rivalry's affect.
But in a double battle, if the opposing Pokemon are male and female, I think the foe you attack determines Rivalry's affect.
Hope I helped. :)
Source