According to Showdown, Magic Bounce activates first.
Note: Showdown aims to be as close to game mechanics as possible and they're almost always correct, but not 100%. I didn't find much in any infroamtion pages about either Magic Bounce or Prankster having specific priority. so my assumption is that the game always takes into account ability before typing (opponent uses move, checks ability modifier, then checks typing modifier). This is also why Rhydon's Lightningrod ability activates when it gets hit by electric attacks even though it would have been immune regardless.