Darwinia and Defcon developer Introversion stopped working on Subversion nearly a year ago, Eurogamer has discovered.

Subversion resembled a collection of tech demos but "the core game itself wasn't really that great", creator Chris Delay told us.

"What happened was, we were tinkering away on Subversion for ages and ages, trying to make some progress - and we did make a lot of progress on it - but we always had this worry that the core game itself wasn't really that great. There wasn't really much to do.

"We had literally years of tech demos but no sort of cohesive core game. It's the core game that was missing.

"Whenever we stopped and thought about it from a high-level point of view, in terms of what's the player going to be doing, where's the fun going to come from, we were drawing a bit of a blank, all the time.

"We finally faced up to it," he said.

Subversion, an infiltration and espionage game where gamers controlled a team of operatives moving through hostile high security buildings, is officially now on hold.

Despite the advances of the past decade, from physics engines and motion control to near photo-realistic graphics, there is one area in which games still have huge scope for improvement. Why, after all this time, are so many videogames still so bad at telling stories?

True, there are more examples of better quality writing to be found these days. More adult themes and activities have crept into titles like Heavy Rain. But there's no escaping the fact that for the most part, most games have about as much narrative sophistication as a Choose Your Own Adventure book.

Scripts, voice acting and the range of choices available to the player have all improved, while open-world games offer a sense of freedom which exists outside of the boundaries of plot arcs. Yet in some ways, these advances serve only to highlight the jarring nature of what happens when you are funnelled into pre-scripted paths.

Why is this? Will it ever be possible to play a game with causality, where you can truly affect the outcome of a story? Eskil Steenberg, the solo developer behind the innovative first-person MMO Love, certainly thinks so.

"It's already been done, except we don't think about it as story-telling," he explains.

"Take Counter-Strike, for instance. You wouldn't call that a strong story-telling game, but its a game where most of the players have stories from the game. It's a very limited story, which involves mostly bombs and hostages and how many people are left. But they are stories and they are told by gamers."

This concept of players developing their story within a set of rules is known as "emergent narrative". Indie games like Dwarf Fortress, the oft-mentioned Minecraft and Steenberg's own Love are leading the way in this field. By offering up gameworlds you can interact with on a deeper level, they create the potential for dynamic, player-authored storylines.

"The human mind is hard-wired to construct narratives as means of explaining what one experiences in one's real life, when watching a movie, or when playing a game," explains Mark Riedl, an assistant professor at the Georgia Institute of Technology's School of Interactive Computers. He is currently conducting research into intelligent narrative computing.

"But while an experience – in real life or a game – can be compiled into a narrative, there is no guarantee that that narrative will be a 'good' one.

"Emergent narrative is simple to achieve; all you need is a rich environment and a good set of rules with which to simulate the microworld," Riedl continues.

"The alternative, which I refer to as "managed experiences", relies on a storyteller that is looking at the microworld, the player, and future possible narrative trajectories, and attempting to enforce some sort of structure."

Steenberg has adopted a similar approach for Love. "The game itself generates the entire world. The players can build a settlement anywhere in the world, and the AI are independent AI characters," he explains.

"There are five different tribes: they fight the players, they help the players, they do all kinds of things – they act as if they are independent actors. And that creates a story that is very very dynamic and a lot of things can happen."

This strategy is markedly different to the one most developers employ today. "Currently I think that games are kind of depressing," says Steenberg. "If you play the first Zelda game – it's 25 years old, but you can do more things in that game than most games you can play today."

He adds: "That tells me we haven't really gotten very far. The games that are closest are games like Fallout, but they are very scripted. They are sort of brute forcing it. Instead of making a roller-coaster, they're making a roller-coaster with multiple tracks and various places where you can switch tracks."

Chris Delay of Introversion Software (the studio behind indie classics Uplink, Darwinia and Defcon) has a different take. Though Fallout 3 is scripted, he feels the game manages to create a sense of freedom with its narrative.

"They did a very convincing job of populating the world. I didn't feel like I was too on rails or anything. It certainly felt much more open than the average first- or third-person shooter," he says.

"So in that sense, they successfully did it. They created a whole world and filled it with interesting stuff. The world never felt to empty or sparse, there was always stuff to do."

Riedl argues that true agency – the freedom to change the world and plot – isn't actually necessary for gamers.

"I think this is where game designers have excelled: scripting plot lines to create the perception of agency, without actually enabling the player to significantly change the direction or outcome of the game's plotline," he says.

"Players have what I call 'micro-agency', the ability to affect the simulated world from moment to moment, but not the ability to affect the overall plot."

But this approach creates a huge burden on game designers, according to Riedl. "Theoretically, for every branch point the amount of plot-related content that must be authored at least doubles, resulting in an exponential growth of plot-related content."

As the sole coders on their games, this is an issue Steenberg and Delay are only too aware of. "It's very, very expensive to produce all this content – its only an option for a really big company," says Delay.

"And of those really big companies, very few choose to go that way, because it's actually much easier and in many ways more reliable just to churn out a Call of Duty: Modern Warfare-style scenario, where generally everything is on rails."

The cost of development means indie developers are frequently more reliant on procedural content – that is, content generated by the game itself.

Delay's latest project, Subversion, is a Mission: Impossible-style game in which the player pulls off heists. The game is set in a rich, procedurally generated gameworld that simulates entire cities, including the interior and exterior of buildings, and the security systems that protect them.

"The typical way to make a heist game would be to design some levels and in those levels to script specific things, like this button switches off the security and this guard walks this path and so on," he says.

"But then you get a predictable result and you get predictable gameplay. Rather than coding specific events and set pieces, I've been programming the world to be as general as I can – so that security systems genuinely work.

"If you have a camera in a room it doesn't just automatically trigger the alarm – it has to be wired into a monitor somewhere, and there has to be a guard watching the monitor and when the guard sees you on the monitor, he has to press the alarm button. And any one of those is a fully simulated system that you can tinker with to your own advantage."

It is this rich simulation of gameworlds which will afford players the option of a wholly dynamic approach to heists. They can approach missions as they choose, and because the world is so deep, numerous options for emergent narrative will exist.

"My initial design idea was to have a gameworld in which there were procedural missions wherever you looked. In any building, anywhere in the city there would be things that you could steal or break into, but that wouldn't be the core game," Delay says.

"My plan is to have this rich gameworld in which the game of Subversion will be set, and the core game will make much more use of hand-crafted levels."

So if well-simulated gameworlds provide scope for emergent narrative, where does this leave the traditional plot? When it comes to RPGs and adventure games, we're used to narrative arcs, plot twists and character development – complex structures that emergent narratives so far don't produce.

According to Riedl, procedurally generating plots in games may not be so far away. "I think procedurally generated side-quests could be low-hanging fruit, in the sense that many of the technologies are ready to go and can be performed fast enough for games to realistically use," he says.

Riedl has already embedded his branching story generation algorithm into the Unreal Tournament engine, as part of work that is partly funded by the US military. By simulating a marketplace, the system challenges the player to prevent a terrorist bomb attack through talking to NPCs. If the player thwarts an NPC's efforts, the algorithm rewrites the narrative - dynamically changing the responses of the NPCs to the player's actions.

"Once one game demonstrates its effectiveness, demand may follow. There will always be room for emergent narrative, when used properly," says Riedl.

"Procedurally branching game narratives, in which a procedural content creation system makes small adjustments to the main plotline of the game, or weaves emergent sub-plots into the game, may come a bit later. In some respects, Mass Effect 2 has taken a step in this direction."

Riedl suggests that similar approaches could be used in multiplayer games, either to co-ordinate NPCs or to co-ordinate player stories when teams break up. However, he's unable to elaborate further on his own plans in this regard due to having signed a non-disclosure agreement.

For Delay, there are issues with dynamic plots that go beyond the purely technical. "There's a a wider question – is it possible for a videogame to tell a good story, in the traditional sense? Because I'm actually not sure," he says.

"Stories tend to be quite character driven. The story emerges out of the characters doing what they would do in the world. And those are the really great stories. But in videogames you've got this massive problem – the main character is controlled by the player. So it just doesn't have any of that inner emotion or drive in the same way that a screen character would do.

"If you then ask, is it possible to procedurally generate a good story, I would say it's even harder. You're starting from a medium that isn't that great at telling stories in the traditional sense, and you're trying to do it electronically."

While Riedl has had success with story generation algorithms, he says developing emotion in the output is challenging.

"In academic research labs such as my own, we have AI systems that can generate short fairy tales from scratch and AI systems that can generate branching story trees for games and training simulations," he says.

"This is because the structures of fairy tales and many computer game plots are relatively straight-forward, focusing on action and causality. The harder problems involve the conveyance of nuance and emotion through story."

So what does the future hold when it comes to story-telling within games? Steenberg and Delay are in agreement: creating the deep, simulated gameworlds needed for well-developed dynamic and emergent narratives is incredibly complex, and success is hard to measure.

"Right now, if I were a commercial game developer, I would be very hesitant to make the kind of game that I've made, for those reasons," says Steenberg.

"On the other hand I'm sort of convinced that at some point, and I think some of my players are at this point, people will play a game and they will understand what a game can be, and how dynamic a game can be.

"And that will make other games feel incredibly old."