It depends on the nature. Basically, the Pokemon likes the flavor of berry that lines up with the stat that is raised, and the Pokemon dislikes the flavor of berry that lines up with the stat that is lowered.
Spicy Cool Power Attack
Sour Tough Stamina Defense
Dry Beauty Skill Special Attack
Bitter Smart Jump Special Defense
Sweet Cute Speed Speed
Here's an example.
If a Pokemon has an adamant nature, it will like Spicy foods, but will dislike dry foods. So if you gave it a Wiki Berry, it will become confused because it hates dry food.
So yes, they are avoidable.
Hope I helped! Source