Yes. In fact, PokeTips recently uploaded a video about it today! You can do it with only 2 badges. Here is the link https://www.youtube.com/watch?v=gOf-2zZkj38
Source: The video above.
NOTE: If you're asking about fighting the Elite 4 and not getting to the Hall of Fame, just tell me and I'll close this answer. Sorry if that's what you meant.