Essentially, my Sableye tried to use that on a Tyranitar in the Battle Tree and it said, "It doesn't affect Tyranitar", and I wanted to know why because it shocked and confused me since I've never seen that happen before.
This is a new mechanic for Gen 7, where Dark type Pokemon are not affected by Prankster, so for example, if my Klefki tried to use Thunder Wave on an Incineroar, Incineroar wouldn't get paralyzed because it has the Dark type
Dark-type Pokémon are now immune to opposing Pokémon's moves that gain priority due to Prankster, including moves called by moves that call other moves (such as Assist and Nature Power) and excluding moves that are repeated as a result of Prankster-affected Instruct. Ally Dark-type Pokémon are still affected by the user's status moves.
Source