The amount of damage you take from an attack ends up proportional to HP x Def or HP x SpDef for special attacks.

This means that you usually want to improve the lowest stat. Here's a bit of math: say for example you start with HP=100, Defense=40 and you add 60 points from EVs.

Adding them to HP means you end up with HP=160, Def=40.

160x40 = 6,400 defense from physical attacks.

Adding them to Defense means you end up with HP=100, Def=100.

100x100 = 10,000 defense: much higher!