Payback will hit regardless if the user took damage beforehand, but it will have 50 bp instead of 100. Payback will not double in damage if the user did not take damage first, so if a move had no effect and didn't hit, it counts as not taking damage and the move will strike with only 50 bp.
In other words, Ursaring will use Slash but the move will fail. Gastly will use Payback, which does hit, but has lower bp than it would if Gastly had taken damage before it used the move.