Ubers
First of all, seeing as both are banned to Ubers by smogon rules I will take my first approach with this being ubers.
Both Mega Gengar and Mega Mawile were great Pokemon in XY Ubers, but with the advent of ORAS, Mega Mawile has suffered quite a ton in the hands of the risen titans and isn't as good as it was before as it is now very susceptible to one of the most common Pokemon in the tier, Primal Groudon, making it a sub-optimal choice when building an Ubers team, however not all hope is lost as you could always try and be innovative and build a Trick Room team around it, if you are building for Ubers.
However, if you would ask me what Mega you should pick between the 2, in nearly every scenario my answer would be Mega Gengar. Its great speed, special attack, coverage and ability to trap annoying Pokemon and kill them off for its team are still as keen as ever and make it one of the best megas in the tier. This obviously makes Destiny Bond a near mandatory move on most sets, that doesn't really matter, as it uses it till very great success.
WiFi Battling
If you're using it for online WiFi battling then it doesn't even matter at all in which playstyle you're using any of them. The only difference would be that Mega Gengar functions well in every playstyle and Mega Mawile in every playstyle bar Stall.
Mega Mawile is much easier to use than Mega Gengar however, as it requires keen skill in prediction to use it till its greatest capacities and enable it to slowly trap and kill or PerishTrap your opponent's Pokemon.
The competitive world in online WiFi battling severely deters from the one created on smogon, due to there being no ban on certain Pokemon, such as Mega Lucario, Mega Kangashan, Blaziken (Mega), Mega Mawile, Aegislash etc., because of this, the competitive environment is roughly like the way it was in early XY OU, in which Mega Mawile wasn't that common at all, because it had a very hard time dealing with all these powerful Pokemon and they just had the tendency of outshining or easily destroying it, whereas Mega Gengar was one of those Pokemon that was doing all the destroying.
According to ranking taken from the official Pokemon site, Mega Gengar is currently the 2nd most common Pokemon in battlespot singles and the most common Mega in it as well, due to it having a much easier time in the current competitive metagame.
However Mega Mawile's ability as a sweeper surpasses those of Mega Gengar. Due to the added threats in battlespot singles, its ability as a sweeper is most impressive under Trick Room, and it still makes a formidable attacker outside of it.
Despite Mega Gengar's huge special attack stat, it's potency as a sweeper is slightly weaker then that of Mega Mawile and generally shines as a trapper and special attacker over the likes f a sweeper, however it is one of the only special attackers who doesn't need any move that can indirectly hit specially defensive walls hard as it has the ability to trap and beat them regardless. With Taunt, it shuts down Chansey and can wear it down with Sludge Wave, as well as beat Sylveon and Clefable with Sludge Wave as well and can thus hit hard at a consistent rate without having to worry about Intimidate or burns, while also being immune to Toxic and potentially hiding behind a sub.
Conclusion
At the moment, I wouldn't be surprised if it seems that the odds are in Mega Gengar's favour, because in singles, it does tend to outdo Mega Mawile, especially in battle spot singles as its flexibility to work its way around so many things is just godlike, however you musn't let this stuff influence your final choice.
Behind a sub, Mega Gengar is more than capable of 2HKO every Pokemon with its coverage move and easily plowing through teams with just Shadow Ball, Sludge Wave and Focus Blast.
Under Trick Room however, Mega Mawile will easily destroy everything with only it STAB attacks. Outside of Trick Room, it becomes a very threatening Pokemon after just one SD as Sucker Punch will utterly destroy everything at +2, or it can simply attempt to attack behind a sub. Due to me not having enough information containing your team, my recommendation will be in Mega Gengar's favour, however, before you make your verdict you should pick the following Mega after taking the following points into consideration.:
The following points are not of major importance if you are yet to build a team around them
- If your team is already very weak to dark types, but you can't afford switching any other Pokemon on your team, consider Mega Mawile.
- If you do not have any reliable switch-ins into Fairy types, consider Mega Mawile
- If you need to get rid of Fairy types ASAP, consider Mega Gengar
- On a highly offensive team, an Offensive Mega Gengar is recommended over Mega Mawile
- Should your Mega Lopunny switch-ins be limited, consider Mega Mawile
- If you have too many physical attackers and are thus susceptible to Intimidate and burns, consider Mega Gengar
