The only ability that somewhat resembles this situation is Zen Mode, which is only known by Darmanitan. When the user hits below 50% HP, it will change Darmanitan into Zen Mode, which adds 50 Base Stat points to both defense stats.
You might be thinking of the ability Defeatist, which is similar to what you're asking. Defeatist makes the user's, instead of the target's, attacks and special attacks do 1x damage at full HP, but then halves attack and special attack when the user hits 50% HP or less.
Or maybe you're thinking of Multiscale, which if the user is at full HP, attacks and special attacks used against it are halved?
I'm not sure the ability you're thinking of exists otherwise. Hope this helps. :)