Readers rate AI-generated stories more highly when they believe they were written by humans

Readers not only struggle to distinguish between human-written and artificial intelligence (AI)-generated literature, but they also tend to rate machine-generated content more highly when they believe it was written by a human author. That is the finding of research led by Villanova University in the United States and published in the scientific journal ‘Judgement and Decision Making’.

The study provides further evidence of how difficult it has become to identify content produced using AI tools. In creative writing, preconceived ideas about what machines are capable of also appear to play an important role in how readers evaluate texts.

Three experiments involving human and ChatGPT stories

The research involved adult participants between the ages of 18 and 81. The researchers set out to determine three things: whether readers could distinguish a story written by a person from one generated artificially, how they rated the quality and engaging nature of both texts, and which factors might affect their ability to identify their origins.

To conduct the study, the researchers selected three fiction short stories written by human authors and previously published in respected literary magazines or collections. Three corresponding stories were then generated using ChatGPT.

The first experiment involved 1,682 participants. Each person read one story and was given information about its supposed origin. In some cases, participants were told that the story had been written by a human, while others were told that it had been generated by AI, regardless of its actual origin. They then rated the quality of the story and how interesting they found it.

The second and third experiments focused on whether readers could identify the origins of the stories themselves. The second involved 424 people, while the third included 481. In both cases, participants read one human-written story and one AI-generated story without being told who had written them. They were then asked to identify the author of each text.

The researchers also assessed the participants’ familiarity with artificial intelligence platforms and their previous experience with fiction literature.

The impact of expectations

The results were particularly striking when the researchers analysed the ratings given to the stories. AI-generated stories received higher scores than human-written ones for both quality and their ability to engage readers.

However, the effect was even stronger when participants believed they were reading a story written by a human. In other words, the same AI-generated text could receive a higher rating simply because the reader did not know its true origin.

Deena Weisberg, a researcher in Villanova University’s Department of Psychological and Brain Sciences and one of the study’s authors, believes the findings reveal a bias in favour of human literary creation.

According to Weisberg, many people continue to associate creative writing with qualities they regard as uniquely human, such as emotional understanding and personal experience. These assumptions may lead people to underestimate what artificial intelligence systems are now capable of producing.

Identifying AI-generated writing remains difficult

The experiments also showed that participants had limited success in determining whether a story had been generated by AI.

In the second experiment, only 39.9% correctly identified the origins of the stories, a result that was actually below what would be expected from random guessing. In the third experiment, the figure rose to 52%, although this was not significantly different from chance.

The findings therefore suggest that reading a text does not necessarily provide enough clues to determine whether it was written by a human or an AI system.

One factor did make a difference, however: prior knowledge of artificial intelligence.

AI literacy improves detection

Participants who reported greater experience using artificial intelligence systems were better at identifying texts produced by these tools. Having greater experience with literature, by contrast, did not have the same effect.

In the second experiment, each additional point in participants’ self-reported AI experience was associated with a 14% increase in the odds of correctly identifying the author.

The third experiment used the Artificial Intelligence Literacy Scale (AILS), a previously validated tool designed to measure knowledge of AI technologies. In this case, each additional point on the scale was associated with a 33% increase in the odds of correctly identifying the author.

The researchers suggest that familiarity with AI tools may make it easier to recognise certain patterns commonly found in machine-generated writing. These include recurring sentence structures and the frequent use of specific stylistic devices, such as em dashes.

For Weisberg, the findings support the need to improve AI literacy, helping people navigate an environment in which machine-generated content is becoming increasingly widespread.

Clarity may work in AI’s favour

The study also supports previous research showing that people can prefer AI-generated writing across different formats, including short stories, poetry and essays.

One possible explanation is the amount of mental effort required to process the content. According to the researchers, AI-generated writing tends to be clearer, more direct and easier to process, whereas human authors may rely more heavily on subtlety, ambiguity or complex narrative structures.

In the stories examined in the study, for example, the AI versions tended to present the central themes more explicitly. Human-written versions, meanwhile, left more room for readers to infer meaning from the characters’ actions and dialogue.

This could help explain why some readers prefer artificially generated texts: they require less mental effort to understand.

The researcher also points out that this phenomenon may be linked to trends that existed before the widespread adoption of AI, including declining attention spans and the consumption of short-form content promoted by digital platforms.

From this perspective, AI may be reinforcing an existing preference for content that is easy and quick to process rather than creating that behaviour itself.