The amount of damage you take from an attack ends up proportional to HP x Def or HP x SpDef for special attacks.
This means that you usually want to improve the lowest stat. Here's a bit of math: say for example you start with HP=100, Defense=40 and you add 60 points from EVs.
Adding them to HP means you end up with HP=160, Def=40.
160x40 = 6,400 defense from physical attacks.
Adding them to Defense means you end up with HP=100, Def=100.
100x100 = 10,000 defense: much higher!