When a Poké Ball hits the Pokémon, as long as it is not deflected, the Poké Ball will open, convert the Pokémon to a form of energy, pull it into its center, and close. A Pokémon in this state is given a chance to struggle to attempt to break free from the ball and escape, being instantly re-converted from energy into matter.
As you may have seen in the anime, Pokémon turn in to a red energy form(Different animation in the games) and return to their Poké Balls, and turn back in to solid life forms when they exit. So, I would assume that their body mass does not change, but its shape and form does, making it easier to store and carry around.
How does this work? No real explanation, there isn't really a confirmed theory, only Game Freak can tell us, and even they probably don't know, it's just something invented to allow trainers to easily transport Pokémon.