It's based on your speed compared to the opponent. So using Gyro Ball against a fast opponent is way more powerful that against a low opponent.
Graveler and Rhydon are fairly slow as well so it won't do much damage (if they were the same speed it would only be power 25). When you have Gyro Ball it's best to make yourself as slow as possible. You would do this with a Speed-hindering nature and no speed EVs.