There are certain conventions, in both literature and life in general, that we follow because they’re just the way things are done. The weight of tradition can be heavy. And sometimes, we prefer it that way. We don’t want to order a burger at McDonalds and discover they’ve decided to start making them out of alligator meat with our first bite (sure, it might be delicious for all I know… but I’d rather know about it before I start eating).
In the world of fiction, there are certain tropes and archetypes that we expect to encounter. Fairy tales always start “once upon a time” and everyone lives happily ever after – obviously, we don’t count the villains. Things happen in threes. Events are signposted ahead of time so we can go “aha!” when they get revealed. And it should be obvious who our heroes and villains are from the moment we meet them.
And all this is fine – but it can get a bit… boring. Nothing ever changes. After a while, the patterns become obvious and the stories become predictable.
Hence the twist. At some point in our stable, predictable stories, there’s a moment when everything flips upside-down. The heroes become villains, or vice versa. The fairy tale kingdom is a lie. The quest is not the quest. This is the key point in any story.
And we’ve grown used to it. The twist in the story is now part of the pattern it’s meant to disrupt. We expect the twist and get annoyed if it doesn’t happen.
This is clearly a problem. How do you include a twist that surprises people when they’re waiting to be surprised?
You need to include a bigger surprise. You need to break convention some other way.
One option is to have two twists. People expect the first. Let them have their moment. And then throw in another one they aren’t expecting. Of course, you still have to follow the rules. The clues for both twists must be laid down, or they’ll feel like you’ve cheated them.
Another option is to change some other aspect. Plot twists are only one possibility. Why not twist something else? A new narrator or main character halfway through the story (Isaac Asimov’s Foundation, I’m looking at you). Or interrupt the story to start a completely new one (a trick that David Mitchell’s Cloud Atlas pulls off five times in a row, before it then closes them all off in reverse order). Or subvert the genre entirely (something Terry Pratchett’s Discworld series did to the fantasy genre repeatedly).
Just remember not to overdo it. If your twist becomes too commonplace, you could end up like M Night Shyalaman – once a master of the surprise ending, he did it so often that people ended up looking for the twist from the start. The magic only works the first time – after that, the magician needs to show us a new trick.