How good is ChatGPT at writing fiction?

Right now: Decidedly mediocre. But that might not last long...

ChatGPT, built by OpenAI, is probably the most sophisticated natural language processor out there. Its most recent update has been all over my social media feeds (though admittedly my social media feeds are not representative of the public at large). I’ve played around with it to see how close I think it is to being able to write fiction.

The short answer is: ChatGPT can already write BAD fiction. Whether that means AI is 1, 10, 100, or infinitely many years away from being able to write GOOD fiction is an open question.

In one example, I asked it to improve on a scene of dialogue among my three main characters. In terms of my own process, this is very clearly first-draft material: It’s got the core concepts that I want to get across, but it’s very much an “info dump” still — when editing I would want to include more indication of setting, more characterization in the form of emotional reactions, and much more interweaving of dialogue with so-called “beats” of action.

Me, prompt #1:

Here's some dialogue I wrote for a scene in a novel. Can you edit it to make it more descriptive of the setting and what the characters are physically doing?

“How DO you know what the Canadian Council of Elders will be discussing next week?” Shehla finally asked Amir when he reported back to the team that the Council was considering trade sanctions against Russia.

He smiled maddeningly, “Oh, I have my sources.”

Later she cornered Winona.

“Amir and I first met more than a decade ago in the Northern Hemisphere Youth Council,” Winona explained. “Every six months, our cohort of 30 up-and-comers from across the North spent 2 intense weeks together in a different country’s capital. You know him — he gains people’s trust practically overnight.

“Everyone was between 18 and 25 years old when the cohort started, but he and I were among the youngest members. So most of those people are in their 30s now, in important jobs. I still write to some of them, but since I decided to be a scientist and most of them work in politics, it’s more about personal updates like who’s getting married, who’s having a baby. I don’t know how he finds the time, but somehow it seems like Amir keeps up an active correspondence with practically everyone.”

ChatGPT Response #1

Here’s a screenshot of the response from ChatGPT. Below the image, I’ve copied the text and marked it up to show what the AI added and deleted.

Here’s the same text copied and marked up with bold to highlight new phrases or sentences it added in, and strikethrough to mark phrases or sentences it cut.

Shehla and Amir were standing in a bustling conference room, surrounded by a team of other young professionals. Shehla looked at Amir with skepticism as he reported on the upcoming discussions of the Canadian Council of Elders.

"How DO you know what the Canadian Council of Elders will be discussing next week?" Shehla finally asked, her arms crossed in front of her chest Amir when he reported back to the team that the Council was considering trade sanctions against Russia.

Amir smiled maddeningly, his eyes sparkling with secret knowledge. "Oh, I have my sources," he replied, tapping the side of his nose with his index finger.

Later, she Shehla cornered Winona, one of their team members, and asked her about Amir's seemingly endless network of connections.

“Amir and I first met more than a decade ago in the Northern Hemisphere Youth Council,” Winona explained. “Every six months, our cohort of 30 up-and-comers from across the North spent 2 intense weeks together in a different country’s capital.

Winona and Amir had first met more than a decade ago, during their time in the Northern Hemisphere Youth Council. Every six months, the two of them, along with 28 other young up-and-comers from across the North, spent two intense weeks together in a different country's capital. [The content is the same, but ChatGPT converted it from dialogue to background exposition.]

"You know him - he gains people's trust practically overnight," Winona explained, a fond smile on her face. "We were all between 18 and 25 years old when the cohort started, but Amir and I were among the youngest members. So most of those people are in their 30s now, in important jobs. I still write to some of them, but since I decided to be a scientist and most of them work in politics, it's more about personal updates like who's getting married, who's having a baby. I don't know how he finds the time, but somehow it seems like Amir keeps up an active correspondence with practically everyone."

Further ChatGPT iteration

I thought ChatGPT did a directionally good job with the first half. In some cases, the prose itself is somewhat stilted and does a lot of “telling” rather than “showing” (e.g., “Shehla looked at Amir with skepticism” is quintessential telling of emotion.) In others, I think it did quite well — e.g., I liked this addition as a way of showing a sense of playful secretiveness: “tapping the side of his nose with his index finger.”

But ChatGPT seemed to forget about my instructions by the second half. So I went through a couple more rounds of prompts, and ended up with this output to replace the last two paragraphs shown above:

ME: Regarding the last thing you wrote, can you take these last two paragraphs and break them up into a dialogue between Shehla and Winona? Add in details about Shehla's emotional reactions. Also, add in that Winona's decision to become a scientist made her family unhappy.

ChatGPT:

Interestingly, it dropped some of the important information about Amir, though that might have been a result of the way I wrote the prompt. It also did split Winona’s monologue into dialogue, but it put all the emotional reactions into the dialogue. Again, telling rather than showing. E.g., a character can tell the other character and the reader “It’s something I’ve always been passionate about.” But the character could also show their passion by, for example, a big smile lighting up their face, and let the reader infer the rest. The latter would almost universally be seen as “better” writing within the book industry.

How would I, Taren, improve on these drafts?

If I were trying to take my draft and polish it, I think ChatGPT’s drafts could be good at sparking some ideas for me or getting through writer’s block. But they’re still decidedly mediocre — not something I could submit to an agent or publisher for consideration.

Here’s a stab at how I might take the combination of my drafts and ChatGPT’s drafts and turn them into a Draft 2 or 3. I’ve bolded where I kept phrases fairly directly from ChatGPT — you can see that there’s not much in that category:

Shehla shifted her weight to the other leg. Hard to get comfortable standing throughout an hour-long meeting. She’d given up her chair since she was the one who had booked the wrong room, thinking this was a meeting of just the Core Group, rather than open to all dues-paying movement members.

Amir’s update sent a murmur through the room: The Canadian Council of Elders was going to discuss trade sanctions against Russia at their upcoming meeting.

Shehla frowned. There hadn’t been anything about that in the news.

"How DO you know what the Canadian Council of Elders will be discussing next week?" Shehla finally asked him after the meeting, her arms crossed in front of her chest.

Amir smiled maddeningly. "Oh, I have my sources." He tapped the side of his nose with his index finger.

Shehla threw her arms in the air. “You’re the worst.”

Later, she cornered Winona.

“You’ve known him forever, right? How does he know all this stuff?”

“Yeah, we met at the Northern Hemisphere Youth Council.” Winona smiled. “We were the babies of that cohort, both of us just 19 years old.”

"Ok, but you don’t have the same sources he does.” Shehla tried to tone down the irritation in her voice, with only mild success.

“You know him. He gains people’s trust practically overnight. I mean, I stay in touch with some of those folks, like who’s getting married and who’s having a baby. But somehow he keeps up an active correspondence with practically everyone. A lot of them are in really important jobs already.”

Shehla balled her fists in her lap. She was more of an insider than she’d ever been, but the layers between her and the center of the onion sometimes still seemed infinite. “Who were they? How did people get into this magic Youth Council?”

“It’s billed as a meritocratic leadership development program. But of course it just so turns out that most of the fellows are from elite political families.” Winona frowned as she started jiggling her leg under the table. “Ha. Like me. I’m sure I only got in because of my dad.”

Shehla softened. “But you’re a scientist, not a politician.”

“Yeah, I am now. And whaddya know, I’m also now the black sheep of the family!” Winona laughed, but it seemed forced. “Anyway, Amir didn’t have connections through his family. He got into the Council through sheer power of charisma and force of will. And now he leverages the network more effectively than anyone else.”

Will AI take all our jobs?

This round of AI won’t take novelists’ jobs. It might make “our” “jobs” (though it’s very presumptuous to assume I might ever get paid to write!) easier: I do think that it’s close to being able to help a human author write better and/or faster. And there may be ways for a novelist to get help from this version of ChatGPT that I haven’t figured out yet.

After this and the other experiments I ran on ChatGPT, though, I’m ready to make a bold prediction. Earlier I posed the following question:

I’m ready to put myself in the 10-year-order-of-magnitude camp: I predict that between 5 and 20 years from now is when some of the “best” — best-selling and best-reviewed — novels in the world will first be written primarily by AIs, with only as much human input editing as human-written bestsellers get from people other than the author.

What do the rest of you think? Also, if you have other ideas for how to use a program like ChatGPT to help someone like me write more/faster/better, let me know, I’d love to try them!