AI Survey Drafts: Give Each Question One Job

Imagine finishing a writing workshop and receiving this question: “How clear and inspiring did you find the session?” You understood the instructions but felt uninspired. Which rating should you choose? A neat five-point scale cannot resolve the problem because the question combines two judgements.

When using AI to prepare a feedback form, examine what each answer would actually tell you. Fluent wording is only the beginning. The useful unit is a question, an answer format and a decision that can reasonably follow from the response.

Decide what the answers will change

Consider a fictional independent author running a paid workshop. She wants to know whether to rewrite the instructions for an opening exercise. Her decision is specific: keep the instructions, clarify them or investigate a particular difficulty. A general request for “ten engaging feedback questions” leaves that purpose unstated.

Her brief can identify the audience, the exercise, the version of the instructions and the proposed decision. She can then examine every suggested question against that brief. A question about refreshments might be reasonable in another survey, but it does not help this decision.

If the uncertainty is more basic—what problem prospective customers experience—start with a customer interview focused on the problem. The conversation can help establish what deserves a structured question.

What AI review research actually tested

There is research on AI as a questionnaire reviewer. In a preprint posted on 10 September 2025, Erica Ann Metheney and Lauren Yehle examined feedback on 262 survey questions. They varied GPT-3.5 versus GPT-4 and whether prompts assigned no persona, a survey-design expert or a linguist. The outputs were collected in 2023.

The feedback included recognised questionnaire-design concerns. However, models also flagged issues that a single human expert did not identify. Such disagreements could include useful additional insights or spurious concerns; they do not establish that every suggested correction is necessary.

This was an evaluation of review feedback, without a test showing that revised questions improved respondents’ answers. It also provides no current performance guarantee for a particular assistant. My practical inference is to treat its comments as proposals requiring a reasoned decision.

Split the judgement

The UK Government Analysis Function’s questionnaire-design guidance, published on 14 March 2023 and currently marked as under review, recommends separating questions that ask for two pieces of information.

In our workshop example, clarity and inspiration need separate questions if both matter. Someone can understand an exercise perfectly while disliking its theme. Another participant might love the theme but struggle to begin. A single score conceals those possibilities.

For the author’s immediate decision, a narrower draft is: “How clear were the instructions for the opening writing exercise?” She can leave inspiration for another purpose. Splitting a question does not require adding every resulting item to the form. Choosing which judgement matters is an editorial decision.

Make the answer choices do the same job

Pew Research Center’s survey-writing guidance explains that closed questions need response options covering reasonable answers without overlapping. It also distinguishes answers written in respondents’ own words from answers selected from a list.

For the clarity question, a draft scale could run from very unclear through somewhat unclear, neither clear nor unclear, somewhat clear and very clear. Someone who did not use the instructions needs a separate way to say so. “Did not use them” describes a different situation from a neutral evaluation.

A follow-up can ask which part, if any, needed clarification. A low rating alone does not identify the sentence to rewrite. Suppose, hypothetically, a participant understood the task but could not find the example. The response suggests investigating where the example appears; simplifying every sentence might miss the difficulty.

Give AI a bounded editing brief

A practical brief to try is: “Review this draft question and its answer options for workshop participants. The decision is whether to revise the opening exercise instructions. Identify possible ambiguities, explain how each could change an answer, and propose a revision preserving the intended subject. Say when a concern depends on information you do not have.”

Then make three decisions yourself:

• Keep a proposed edit when it resolves a specific ambiguity without changing the subject.
• Reject an edit that substitutes another question, such as enjoyment for clarity.
• Test an unresolved concern when you need evidence about how participants interpret the wording.

Ask for the reasoning before commissioning another complete rewrite. Otherwise, successive polished drafts can move away from the original decision while appearing increasingly finished. Keep the original alongside the proposed revision so that a change in meaning remains visible.

Watch someone try to answer

The US Census Bureau’s questionnaire-testing guidance, last revised on 8 October 2021, describes cognitive interviews in which potential respondents explain their thinking while answering draft questions. Follow-up probes can explore their interpretation of terms, time periods and response choices.

For a small workshop form, adapt that principle by inviting people with relevant experience to complete the draft and explain a few answers. Ask what they included in “instructions”, how they chose a rating and whether an answer option was missing. These are suggested checks, rather than a validated testing protocol.

Let the wording do its work before explaining your intended meaning. In the fictional example, if someone includes the facilitator’s spoken demonstration while another considers only the printed sheet, record that difference. Decide which you meant, revise the question and try the revision again.

Keep the question with its results

Save the final wording, response options, question order and form version with the responses. For this author, also record which exercise instructions participants saw. A later comparison needs that context: a changed score could accompany changed teaching materials, changed wording or both.

Finally, write down what the answers justify doing next. A confusing instruction may deserve revision and another trial. A favourable clarity rating does not, by itself, settle a decision about the workshop’s price, subject or length. AI-assisted drafting becomes useful when it leaves the author with an answer she can interpret and a specific choice she can make.

Sources

• Metheney and Yehle: Generative AI as a Safety Net for Survey Question Refinement — preprint, 10 September 2025.
• Government Analysis Function: Questionnaire design guidance — 14 March 2023; page under review.
• Pew Research Center: Writing Survey Questions — guidance checked 24 October 2026.
• US Census Bureau: Questionnaire Testing and Evaluation Methods — revised 8 October 2021.