Paybacks power doubles if the Pokémon using it is hit first and then uses Payback right? So would the power double if it didn't acctually take damage behind a substitute?
Payback does 2x damages if you're attacked first. However, unlike Moves like Counter, Avalanche, and Focus Punch, your Priority does not lower (same priority as Tackle, Ember, etc.) to facilitate this double damage. Substitute only affects Focus Punch being interrupted by damage. Payback, Avalanche, etc. work normally and Counter and Mirror Coat are based on the Substitute's HP.
Quoted from GameFAQ