Garchomp-An annoyingly powerful dragon hitting base 600 total stats. He resists stealth rock, gets STAB outrage and STAB earthquake, letting him cover tons of pokemon. A single swords dance can bring his attack power close to 800, not even including an item. A choice scarf can let him go first, even if he loses the swords dance opportunity. A yache berry weakens his ice weakness, so even if you're lucky enough to get an ice hit on him, he can take it like nothing happened. Most of the priority moves aren't enough to take him down, and he wipes the floor with almost anything strong enough to take him down. Throw in a sandstorm from Tyranitar for Sand veil+more residual damage, and you shouldn't be surprised to get a RQ from him. take the situation:
Garchomp+Life Orb+ one Swords Dance+ outrage
If he attacks a Standard Scizor with this, it'll do 280 damage at the least. This is a physically bulky pokemon that resists Dragon, yet it just about OHKOs it. Nothing is able to 4X resist dragon, so one false move, and you're screwed. Or worse yet, you could just baton pass stats to him and have absolutely no risk for Garchomp whatsoever.
Ferrothorn-He won't always cause ragequits due to how common he is, but constant switch ins do annoy people.
Jirachi/Togekiss/sky shaymin-Pretty obvious reasons. Paraflinch is extremely annoying.
durant+entrainment+Chandelure-Entrainment on Durant is annoying enough in itself to bring about a ragequit, but throwing in Chandelure's Shadow tag to stall and set up also brings out the frustration in your opponent.
Stall teams-This includes putting pokemon like Skarmory, Scizor, or Ferrothorn in the rain to soften their fire weakness, sandstorm Shuckle, bulky pokemon, whimsicott, sableye, anything. Stalling with moves like toxic and substitute, while just stalling out with status and residual damage annoys the opponent. Even if they can get in some damage, just recovering is a giant spit in their face.
battling outside your tier-For example, challenging a UU trainer with an OU team will often result in them quitting, assuming they even accept in the first place. They didn't build their team around other groups, like OU pokemon, their team is geared towards countering UU threats. An OU trainer wouldn't want an Uber trainer challenging them, so that's how BL, UU, and NU trainers feel. If they want to battle upper tiers, then they'll challenge you.
Hazards+Phazing-Using entry hazards, then just spamming whirlwind or something is just plain annoying. Don't do it.
blaziken stall-Setting up speed boosts on him, then passing it to a slow, strong pokemon with priority is not fun for the opponent.
chandelure trapping-With a proper switch in, Chandelure can pretty much destroy you. God forbid if you had a choice item, but it's not like those are used often competitively.
instant weather-Pretty self explanatory. Powered pokemon, swift swim or chlorophyll, residual damage, all from sending out one pokemon? Not fun. The main way to counter weather teams? Use weather yourself, meaning you can either centralize the game around 4 pokemon (3 since hail is considered the weakest in OU) or waste a moveslot carrying sunny day, rain dance, sandstorm, or Hail and just hope that the weather you come up against isn't the same one.
stat boosters-This one is extremely annoying for a lot of people. Setting up all the time gets really annoying, as can baton passing.
If someone knows they're gonna lose a battle, they'll quit. It's not a matter of disrespect to you, but why would you sit through an annoying battle you know you're gonna lose. If someone quits, just take it as a free win and move on. Those are a few I could think of. If I remember more, i'll add it. Everyone has their own definition of annoying though.