There's no known percentage for this, but here are a few factors I've seen:
- "level"; by this I mean how many times you've seen the Pokemon, indicated by a number on the top right when you tap into the Pokemon on your Dex Nav. The higher the number, the less likely you are to fail. Around 14-15 is when your Dex Nav start getting more reliable. Before that, you're likely to fail in your search.
- grass; usually you should be outside the grass for the search to succeed. While in grass, it doesn't seem to work as well (or at all in my experience). Also the more grass, the more likely a shaking patch will generate.
- failing to get to the Pokemon; if you're chaining with the Dex Nav and you fail to defeat/ capture the Pokemon you've searched (so you leave it for too long, run away from it or scare it away etc.), your chain will be broken and the Dex Nav will fail to search the Pokemon. The only way I know to solve this is resetting (exit & re-enter the area).
- steps; the Dex Nav seems to be a bit similar to the Radar in this sense, as you need to run around a bit before it'll work again after a failure. As for the amount of steps needed, not sure... but considering the Radar needs ~50, I would guess around the same.
There's also resetting the Dex Nav, by exiting the route/ area and re-entering it, which will reset your searches and the success rate.
If/ when someone posts a definitive answer I'll hide this, but the above is as far as I know/ seen as of yet.
