Instead of getting an email when someone comments or votes on your answer, it should be a wall post. If not, then at least provide an email notification showing you have been emailed. I think this would make some things easier.
You are only emailed if you tick the box.
If you don't tick the box you can just check the "My updates" link next to your username at the top.
