What would happen if you were hit by both attacks? Since torment forces you to use a different attack, and encore forces you to repeat attacks. Do they cancel each other out, or is it the last one to hit you?
Encore+Torment would simply make the pokemon struggle every alternate turn, because Torment says they can't use the same move twice in a row, while Encore prevents them from using a different move, so they have to use Struggle.
It would fail because just read on. Example: Let's say the opponent uses torment and you use Iron Head. And then it uses Encore then you struggle because you used a move 2 times. The next turn you are fine because you will restart.