I had 3 people, some houses where people lived, pokecenter and a pokemarket, now it's pokecenter and one house where someone lives, also Ken stayed. I have not linked with b/w (I've linked with w2) so people could move. I don't hack.
Sometimes, if you don't interact with the people in Black City or White Forest, they leave. Different people, depending on their names, have a higher chance of leaving.