No, you're not doing anything wrong.
The Destiny Knot chooses 1 IV to ignore when creating the baby each egg cycle, then for the other 5 IVs it chooses between the parents for that IV at random. Since both of your Charmanders have 4 IVs each, you simply have a moderately high chance of getting the non-31 IVs to be chosen. Eventually, you may get a Charmander that is exactly how you want.
It is substantially easier with better breed stock. I obtained a 6 IV foreign Ditto from /r/BreedingDittos that will never leave my daycare ever again. Using that Ditto, I was able to take my random no IV Bulbasaur I had, and breed a new Bulbasaur (came out 4 IV I think), which replaced the random one in the Daycare. That union produced a better Bulbasaur, which replaced it's parent in the Daycare, and so on. I have since churned out a LOT of "perfect" 5 IV Bulbasaurs and perfect 6 IV Bulbasaurs in my quest for a "perfect" Shiny Bulbasaur. Now, I get 5 IVs every time courtesy of the Destiny Knot.
Which IVs those happen to be, on the other hand.... yeah, takes time.