Table of Contents
In 2023, researchers started noticing survey answers that were suspiciously articulate: perfectly structured paragraphs, flawless grammar, and the emotional depth of a press release. Large language models had arrived in the respondent pool. Today the question is not whether AI-generated responses are in survey data, but how many of yours they have already contaminated.
This guide covers the practical signals that expose AI-written responses, a manual detection workflow for small studies, and where automation takes over for larger ones. It is the defensive companion to our AI survey responder guide, which addresses the other side: people tempted to generate fake respondents on purpose.
For a complete quality strategy, pair detection with our guides to collecting high-quality survey data and using qualification checks.
Why people paste AI into your survey
Understanding the motive helps you spot the residue. Three groups produce AI-shaped answers:
- Incentive farmers. Paid panels and rewards attract people optimizing for completion speed. AI writes their open-ends while they watch TV.
- Full bots. Scripted respondents that answer everything, built to harvest incentives at scale. These fail on many signals at once.
- Cornered humans. A real, mostly honest respondent hits your mandatory 200-word open-end, sighs, and asks a chatbot. The closed-ended answers are real; the text is not.
That third group matters: it means AI detection is not just fraud detection. Survey design that respects people's time (shorter open-ends, fewer mandatory essays) genuinely reduces contamination.
The tells in open-ended answers
No single tell is proof. Clusters of them are strong evidence.
- Generic perfection. Flawless grammar and structure paired with zero specific detail. Ask about "a frustrating shopping experience" and AI returns an essay about frustrating shopping experiences in general; humans name the store.
- The assistant register. Phrases like "it is important to note", "overall, I believe", "in conclusion", and hedged both-sides constructions. Real respondents are blunt, partial, and frequently ungrammatical.
- Restating your question. AI loves to open by mirroring the prompt: "One aspect of the product I find frustrating is...". Humans just answer.
- Uniform length and structure. Ten open-ends, each two tidy sentences, across a whole response. Human effort is ragged: one-word answers next to rants.
- No personal anchoring. Missing first-person specifics: no names, places, prices, dates, or petty grievances. Lived experience is concrete.
- Cross-respondent echoes. The same distinctive phrasing appearing in multiple "different" respondents is close to conclusive; it means one operator or one prompt behind several submissions.
Human vs AI, side by side
Question: "What almost stopped you from buying?" Human: "shipping was 6.99 which felt like a scam for a t-shirt". AI: "The primary factor that gave me pause was the shipping cost, which seemed disproportionate relative to the item's value. However, the overall value proposition ultimately justified the purchase."
The signals outside the text
Text tells are where you look first, but the strongest evidence is usually in the metadata:
Non-text signals of AI-assisted responding
| Signal | What to look for |
|---|---|
| Completion time | Long, thoughtful open-ends submitted at speedrun pace |
| Per-page timing | A 150-word answer appearing after 20 seconds on the page |
| Paste behavior | Text appearing in one paste event rather than keystrokes |
| Attention checks | Failed instructed-response items next to eloquent essays |
| Duplicates | Same device or network fingerprint behind multiple respondents |
| Answer coherence | Open-end praising a feature the closed-ends said they never used |
The single most damning combination: essay-quality text plus bottom-decile completion time plus a failed attention check. Any one alone has innocent explanations. Together they do not.
A manual detection workflow for small studies
For a thesis-sized dataset (a few hundred responses), an afternoon of structured cleaning is enough:
- Sort by completion time. Flag the fastest 10% and anything below roughly a third of the median.
- Score attention checks. If you placed them well (see how to write attention checks), failures plus speed is your first exclusion tier. An open-ended micro-check ("summarize the scenario in one sentence") is particularly AI-resistant, and worth adding from the attention check generator before fieldwork.
- Read open-ends in one sitting. Echoes and the assistant register jump out when you read fifty answers consecutively rather than one at a time.
- Cross-check coherence. For flagged rows, compare the open-end story against the closed-ended answers.
- Apply a pre-set rule. Two or more independent flags, exclude and report. One flag, keep but sensitivity-test your results without them.
Document everything. "We excluded 23 responses (6.1%) flagged on two or more quality signals" is a sentence that makes reviewers trust the rest of your paper.
Do not outsource judgment to AI detectors
Public AI-text detectors have real false-positive rates, and they flag non-native English writers disproportionately. Use them, at most, as one weak signal among several, never as a sole reason to throw away a response.
When to automate
Manual review stops scaling around a few hundred responses, and it happens after the bad data is already collected. For ongoing or larger studies the screening should move upstream: timing analysis, duplicate detection, and automated review of open-ended answers. That is the approach behind SurveySwap Protect, built to score respondent quality before a response lands in your export, so cleaning becomes review instead of archaeology.
Prevention also has a sampling dimension. Anonymous links posted into the void attract exactly the incentive-farming traffic that produces AI answers. Recruiting through a reciprocity-based pool like the free survey exchange, where respondents are researchers with skin in the game, cuts the problem off at the source.
Frequently asked questions
How common are AI-generated survey responses?
Estimates vary by platform and incentive structure, but studies of crowdsourced panels since 2023 consistently find measurable AI contamination in open-ended answers, from a few percent to well over a third in unprotected samples. Assume nonzero and design for it.
Can software reliably detect AI-written text?
Not on its own. AI-text detectors produce false positives, especially against non-native speakers. Reliable detection combines text signals with timing, paste events, attention checks, and duplicate detection.
What is the best single defense against AI responses?
An open-ended micro-check tied to your survey's specific content, combined with per-page timing. Generic checks are easy to script past; questions that require processing your actual scenario are not.
Should I delete suspicious responses?
Exclude on multiple independent flags, using a rule you set before fieldwork, and report the exclusions. Deleting on gut feeling after seeing your results is a bias risk in the other direction.
Every quality tool mentioned here lives in the free research tools collection.
