The AI Writer Detection Scam
I ran my writing through ten “human detectors”…then I tested the Declaration of Independence.
After 51 articles on Substack, I wanted proof I wasn’t sounding like a wine chatbot. So I did what anxious writers do these days, I checked it with an AI checker, 10 actually.
All of these are free to do:
I figured I would take the time after writing an article nearly every week for the last year on Substack to analyze if I write like a human or a LLM. This is Substack, where the human writers are, right?
But how can we be sure?
Well, I’m happy to run my own experiment on my own writing. I took two different styles of my writing, the “Rant” style which Paul Gregutt pointed out on my “Intention” article and the “Wiki” style of Paso Bro, which is more a taxonomy of a particular wine archetype.
Using some tips from fellow Substacker Anangsha Alammyan (Explore AI with Anangsha) I tested 10 popular AI writing detectors:
Quetext
Winston
Humalingo
Originality
Quillbot
GetSolved
GPTZero
Copyleaks
Undetectable
Surfer
I input my last 2 articles, Winemaker Intention and Paso Bro, into each of the AI Detectors. Here are the results1:
Humalingo thinks I write like a LLM, declaring me 11% Human. Winston, Quillbot, Copyleaks assign a high probability (100% perfect score) that I’m a Human.
GPT Zero is an interesting result — article to article with the widest spread 77% between 100% human or only 23% human. The rest of the results fall in between.
Paso Bro has a wide range of probability scores. I’m guessing because I wrote it sort of like a Wikipedia entry, which LLMs sample, but that’s a guess.
Intention is a personal rant with changes in tone and perspective. It’s a little messy, emotional, it’s a rant after all, suggesting these detectors like rougher, personal edges rather than clarity and precision.
Humalingo says that “your text shows several issues, an AI-Like tone, repetitive phrasing, uneven rhythm, potential plagiarism, and low readability…” with a suggestion button below SCREAMING “Fix all issues”.
It seems human writing, in Humalingo’s infinite wisdom, is a problem that needs fixing, for a fee of course.
Well Humalingo, I put another document in your AI detector. It’s called the Declaration of Independence of the United States of America…
Houston…we have a problem.
The Declaration of Independence got a lower score than my personal rant on winemaker ‘Intention’. The Declaration got a 9% Human Score on Humalingo.
What are these AI detectors doing, simply causing doubt among writers? Creating a disease that only a “Humanizer” can fix? Convenient.
Sorry President Jefferson, you write like a bot but we can fix it for a fee.
I don’t think I write like an LLM, except, it seems, when I do, the Paso Bro “wiki” style, which statistically on Substack the “Bro” articles are my most viewed/read. Which makes me wonder, why? Easy to read? Funny? Bullet style? Who knows.
I’m writing in a way that pays attention to the world around me. I don’t make-up any of the Bro’s personality characteristics, I observe them in real-time.
Writing has to be interesting after all and I’m not worried about what these AI detectors think about mine. This was not meant to be some exhaustive research project or thinking deeply about the logic/programming language of a detector, this was a quick common sense check on emerging software.
The detectors and AI writing will improve every second of every day, because the LLM is taking laps on every piece of writing every second of every day improving — while human writers might knock out a 10 hour session on a typewriter cause it “connects” them to their writing.
At some point in the not too distant future, 80%+ of people won’t care. In fact, right now, do you care if an AI wrote the latest article on “What wine to pair with the World Cup?” or some boring list of “13 White Wines to Drink this Summer!”
I certainly don’t.
Nobody seems bothered by Google Maps choosing an alternate route for you mid-trip or the robot sorting your package at the Amazon fulfillment center (they have over 1 million of them!) or the human-driverless Waymo in San Francisco. AI/LLMs are here, they’re not going away and more people will use them to their advantage (sad tuba sound) while you grind it out on a legal pad, typewriter, or HP12c.
I’m happy to keep writing this Substack for fun/enjoyment, to work my human brain, for therapy, to share my experiences (rant a little), connect with people about wine, hopefully make people laugh from time-to-time and keep it under 1000 words.
All of these are free to do
Results are captured in screenshots in the Gallery of pictures to see each individual score/result/probability. If you are an email subscriber, you won’t see it in the email only on Substack after it is posted. Why? Too many pictures could land the email in a spam folder.
David, I too got worried about how my writing might be perceived in the age of AI, so after reading your piece I ran several of my essays through a collection of AI detectors.
I was horrified by the results but not because they thought I was an AI. Even worse... some of them thought I was a human.
One detector confidently informed me that an essay built from thirty years of standing in wineries, talking to customers, cleaning tanks, managing inventory, and making wine was only 27% human. Apparently the remaining 73% was generated by a large language model that spent decades hauling hoses and arguing with distributors.
At this point I don't think the problem is my writing....
Like you, I started wondering what exactly these detectors are measuring. If they can mistake the Declaration of Independence for AI-generated text, perhaps they are not detecting humanity so much as detecting statistical patterns they associate with humanity.
The irony is that many of the things we spend years trying to improve as writers, clarity, coherence, consistent sentance structure, fluid prose etc. are the things these tools seem to be flagging as suspicious, which leaves us in the strange position of being told that writing well is evidence that we didn't write it.
I'd like to think that our readers are better judges than software. They may not always know whether a human or an AI touched a piece of text, but they usually know whether the writer has actually seen something worth talking about- or at least i hope they do. For now, I'm willing to trust them more than the robots.
To be fair to the Declaration test - something that sounded that stilted and old-world jargony would likely be written by AI today. Especially if the attempt to mimic the style was so absolute. I'm assuming there was no way to really make the software realize it was an ancient document.