I'm working on a project with a nest ball caterpie. Don't ask.
It inherents its Mothers pokeball.
If the Pokemon doesnt have a mother(if its a ditto) it will go in a regular pokeball.
So if you want a Nest Ball one catch a female in a nest ball and breed it.
Source: http://www.smogon.com/forums/threads/special-poké-ball-inheritance-guide-gp-1-1.3498616/ and expierience
As most things go in Pokémon breeding, it is the mother's Poké Ball that is passed down.