Fact Checked by 
A new study by scientists at the University of Pittsburgh has found that non-expert poetry readers are worse at identifying artificial intelligence (AI)-generated poetry than if they were guessing.
The study asked ChatGPT-4, an AI LLM, to generate several poems in the style of famous human poets (including Plath, Whitman, and Byron) and juxtapose them alongside human-written poems by famous poets (e.g., T. S. Eliot). Participants were then directed to determine whether a poem was written by a human or an AI and to identify which poems they preferred and why.
The study found that the sample group of non-expert readers had a 46.6% chance of identifying AI-generated poems. The participants were also “more likely to judge AI-generated poems as human-authored than actual human-authored poems.”
As a further finding, the study concluded that participants rated AI-generated poetry more favorably in certain areas than human-written poetry. The fact that AI-generated poetry rated higher in areas such as rhythm and beauty likely contributed to the mistaken attribution to human authors.
The study’s authors believe that AI-generated poetry being simpler is a large contributing factor to this unexpected preference. AI poems were more straightforward and participants were much more likely to describe human-authored poetry with some variation of ‘doesn’t make sense’ that they were to use that description for AI poetry (144 vs. 29).
This leads to a “more human than human” effect. Because the study participants preferred AI-generated poems, they assumed they were written by humans because they believed they would prefer poems written by humans.
The controversial realm of AI writing
In their conclusion, the study authors note that this represents a step forward for AI-generated writing. Until recently, it had not been a challenge for humans to identify AI-created poetry – participants could identify GPT-2-created poetry.
Early this year, an AI-generated story won the top prize at a national writing competition in China which was judged by the Jiangsu Science Writers Association. The director of the editorial department of the People’s Literature Publishing House issued a dire warning that AI may begin to threaten the jobs of writers and could cause permanent damage to the literary landscape.
Featured image credit: Midjourney
