4 EVs = 1 stat point only applies for level 100 Pokemon; for every other level, due to how the formula for determining stats works, the effects of EVs will be less noticeable. Basically, the lower the Pokemon's level is, the more EVs are needed to actually effect the stat.
To be clear though, Pokemon that were EV trained at a lower level are at no disadvantage, since this formula is revised every time a Pokemon levels up or gains EVs. The level the Pokemon was at when it gained its EVs makes absolutely no difference at level 100.
Expert edit: The below paragraph is wrong in all games before Generation 5.
Another little detail to throw in is that a Pokemon doesn't get its EV points when it levels up, it gets them as soon as they're earned. This basically means that if a Pokemon gains enough EVs without levelling up (enough to increase the stat value provided as a result of the above formulas), it can gain stat points without levelling up. The stat gains you see when your Pokemon levels up are the extra points added when the value of "Level" in the formula increased and the stat point value changed, with the changes from EVs included.
Credit to Bulbapedia for the formulas.