Does it matter what level I pump some vitamins into my Pokemon, I'm assuming you would do this at the lowest level possible?
You want to try and do this at the lowest possible level, yes. But you see a better effect (Less EVs per stat to raise the stat) at higher levels.
So just do it as soon as you can, as it won’t have any negative impact.