While in a dungeon, if you defeat a Pokémon, there is a chance you are given the option to recruit it.
- The team leader must deal the finishing attack that causes the target to faint.
- The enemy must be on a tile next to the team leader when the finishing blow is delivered.(Pokemon such as legendaries that have a short cutscene after defeat are an exception to this rule)
There are ways to increase the recruit rate, you can see them here or here.
Edit: Getting a Pokémon to open up to the team is much like happiness in the core series. All you have to do is take the Pokémon out to dungeons, and avoid letting it faint, do many things with it, and eventually it will be happy enough to evolve, if it is a Pokémon that evolves by this method.