The reason your Black City is shrinking is because you don't visit it enough. When you don't pay Black City/White Forest any attention, the residents leave, and some of the buildings dissapear as well. You can regain residents/buildings over Entralink. Since you are using Black, you must go to the world of a White owner, and talk to some of the poeple in their White Forest. These people may come over to your Black City to replace the left residents. You can have 10 poeple at Black Forest, no more than that.
PS. This does not work while a mission is active.