How likely is it that I got Moltres down to the lowest HP possible without killing it, then used 20 Great Balls and 15 Ultra Balls and didn't catch it?

Great Balls

You have a 1.234% chance of capturing it per ball. Thus, you have at least a 50% chance of catching it within 56 balls and at least a 95% chance of catching it within 242 balls.

Ultra Balls

You have a 2.106% chance of capturing it per ball. Thus, you have at least a 50% chance of catching it within 33 balls and at least a 95% chance of catching it within 141 balls.

source 1 - ((1 - 1.234%)^20 * (1 - 2.106%)^15) = 0.433 (or 43.3%) You should probably try putting the Moltres to sleep.