The Kalos Elite 4 is one of the easiest in the franchise, with Diantha in particular being infamous for her ineptitude. As her ace is Level 68 (and Elite 4 aces are slightly lower, at 65), you can get away with somewhere around Level 65 to 70, depending on how much of a challenge you want.
This being said, you do have one of the most overpowered Pokemon available: Xerneas. Its raw firepower (even without Geomancy) is enough to shred most of the Elite 4, so you could even go lower than Level 65 if you wanted to pretend like these fights are difficult.