ah, i see where you're confused here.
critical hits remove all stat differences that would go agaisn't the user ( your +6 defense ) when being used, AND do 2X damage.
the critical hit ignored your +6 defense, but when he didn't crit hit you, the +6 defense was still applied.
so yeah =) hope i typed that out clear enough.