Pokemon games check for move successes and failures in a very specific order. This is a list of checks that happen before protean and libero activate: aura wheel, aurora veil, clangorous soul (checks if user has enough HP), counter, mirror coat, metal burst, destiny bond, fake out, first impression, mat block, fling, follow me, rage powder, future sight, doom desire, last resort, no retreat (checks if user has already used no retreat), poltergeist, protect-like moves, rest, sucker punch, sleep talk, snore, steel roller, stockpile, stuff cheeks, swallow, spit up, teleport
Checks that happen after protean and libero activate: attract, venom drench, coaching, dream eater, endeavor, ingrain, leech seed, OHKO moves, perish song, major status inflicting moves (checks if target already has a major status), stat changing moves (checks if target's stats can't go higher or lower), torment, worry seed, yawn
Moves in the second list can still activate protean or libero if they fail. Please tell me if the lists are incomplete or outdated
source