Keep in mind that Pokémon can be based off of multiple themes.
For example, Primarina is both a siren and a seal. Bisharp is based off of a samurai and a bishop (from chess). Pokémon do not have to be as straight-forward as "bird" or "magnet" or "ice cream." Going by that logic, it makes sense for Gardevoir to take inspiration from a few different sources.
Knight's Robes
Notice how the cloth drapes down under the waist, similar to Gardevoir's gown.
Gardevoir resembles a principal dancer. Its exaggerated head with minimal facial expression and an overall frail, featureless physique could also be attributed to anesama ningyou, a style of traditional Japanese paper doll. Its protective nature over its Trainer may suggest that it is based on a guardian angel, though its ghost-like characteristics could also suggest that it is based on ghosts that refuse to leave this world.
Anesama Ningyou
These dolls are unisex, despite their feminine appearance (well, that particular doll is female, but the male ones look similar). They somewhat resemble Gardevoir, especially the blank face and dress.
Principal Dancer
Kirlia resembles a dancer the most, but certain themes carry on to Gardevoir — particularly the form of the dancer's dress being similar to Gardevoir's white areas. While this design is quite feminine, there are male dancers as well (although they forgo tutus for uncomfortable looking tights).
Guardian Angel
Once again, similar robes. Also note that angels, being divine beings, do not have any particular gender. In the Bible they appear as both males and females, depending on how they wish to appear. The main idea is that angels are unisex.
Conclusion
Gardevoir is based off of many different things, most of them unisex with the exception of knights and dancers. Also, it is very important to consider cultural differences when considering Gardevoir's appearances. In Japan, the first thing they think when they see Gardevoir is not (or was not) "girl." This is apparent in Gardevoir's Japanese name (Sir Knight). Where we see a woman-esque figure in a dress, they see a knight or a paper doll. It's not as though Game Freak made a deliberately feminine Pokémon and then decided to make it both genders (although they have done this in the past coughMachampcough).
Postscript: Mega Gardevoir and Gallade
Gallade is a male-only evolution of Kirlia and considered Gardevoir's "male counterpart" despite male Gardevoir still being a thing. While not a concrete fact, the popular theory is that Game Freak realized how the western world considered Gardevoir to be feminine and not unisex, so they made a kickass masculine evolution for Kilria with swords on its arms.
Mega Gardevoir came many years after Gardevoir, when the whole world — even Japan — considered Gardevoir to be a primarily female Pokémon. They already had Gallade to fill the male role. Thus, it makes sense for them to simply role with the two X chromosomes and give Mega Gardevoir a ball gown, masquerade mask, and fancy gloves.