Using AI to analyse open-text survey answers

What AI theming does

A language model reads the answers to one question and groups them into themes, such as price, waiting times or staff. It names each theme, counts the answers that raise it and picks quotes that represent it. Done well, it gives you a first draft of a coding frame in minutes rather than hours.

Where it goes wrong

• Themes too broad to act on, such as “Service”.
• An answer that raises two topics filed under only one of them.
• Sarcasm and mixed feelings read literally.
• Small but important themes merged into bigger ones, or dropped.
• Counts quoted with more precision than the sample supports.
• A quote that identifies the person who wrote it.

How to check it

• Read a sample of the raw answers yourself before you look at the themes.
• Open each theme and check that its quotes really belong there.
• Look at what lands in a catch-all theme: a large one means the themes missed something.
• Check how many answers were read against how many there were.
• Compare themes between groups only when each group has enough answers.
• Treat the result as a draft. The conclusions, and the decisions, stay with you.

What Thisys does

• Nothing is read by AI until you switch analysis on for that survey, and every survey page tells respondents whether AI reads their answers.
• For each open-text question it reads up to 200 answers, sampled evenly across the whole fieldwork period, and the first 400 characters of each. The result says how many answers it read, out of how many.
• It groups them into themes, each with a sentiment label (positive, negative, mixed or neutral), a count and representative quotes. An answer that raises two topics counts in both themes, so the shares can add up to more than 100%, and the theme cards say so.
• Answers held back for moderation review are not read, only counted.
• In a Decision Report, the findings are calculated from the data without AI. An AI sentence whose figures or statements contradict them is left out, never rewritten, and the reader is told how many were withheld.
• Ask your data answers a direct question from the responses, with the price shown before you ask.
• Groups smaller than the survey's minimum reporting group are hidden from the AI, as they are everywhere else.
• Every AI result carries a notice naming what can go wrong and the check that catches it.
• AI is paid for in credits. It runs on Anthropic Claude, the one AI provider, and your content is never used to train it.

A sensible way to work

1. Read 20 or 30 answers yourself.
2. Run the theming.
3. Check each theme's quotes, and rename any theme too vague to act on.
4. Compare themes between groups, such as detractors and promoters, where each group is large enough.
5. Decide what to do, and write each action down with an owner.
6. Run it again after the next round and compare.

Privacy

People write names, phone numbers and health details into comment boxes. Tell respondents if AI will read their answers, keep personal data out of anything you share, and check quotes before they leave your organisation.