Have people never heard of suspense of disbelief?
It's just like how they manage to set up tables, chairs and cookware in the anime despite only carrying a medium-sized backpack around with them.
For what it's worth, I always imagine that the held item is invisible and in another dimension, so it doesn't get damaged, nor does the Pokemon actually need to hold it. It's a good a way to visualise/ set your mind at ease as any.
