Well, it is brought up a bit after he returns that she rejected his love, as did other girls as we see through the series. He spent a while with Professor Ivy. So after all he did for her and all the time they spent together, when she rejected him, I'm sure it put him in a horrible state of depression. And when people bring her up, I'm sure it's a huge reminder of what happened, which is why he doesn't like when people bring it up.