AI-Written Onboarding: Test What a Newcomer Can Actually See

A ceramics studio gives its first freelance assistant a guide to logging orders. The owner asks AI to tidy the wording, and the revised guide starts: ‘Open the board, select the right batch and release the order.’ Every sentence sounds clear to the owner. On the assistant’s screen, there are three boards, two batches with similar names and no release button. The guide reads well because the owner can supply the missing information without noticing.

When you ask AI to rewrite a process you already know, separate clarity of language from completeness of instruction. Shorter sentences can help, but they cannot tell a newcomer which board you mean unless that information is present or verified. The most useful review starts where the newcomer will start, with their account, their permissions and the information they have actually received.

Why familiar instructions can seem more complete than they are

In their 1989 experiments on the curse of knowledge, Colin Camerer, George Loewenstein and Martin Weber studied what happens when people with additional information try to reproduce the judgements of people who lack it. The informed participants’ judgements were influenced by information that the other participants did not have. These were economic experiments, not tests of workplace guides, but the distinction is useful when you write for someone at a different starting point.

A separate 2000 study by Boaz Keysar and colleagues tracked eye movements as people followed spoken instructions to move objects. Listeners sometimes considered objects that the speaker could not see before using shared information to resolve the reference. The study is about interpreting instructions, not authoring them. Together, the two pieces of research suggest a practical question for a guide writer: what will this instruction point to from the reader’s side?

The application to AI is an editorial inference. A language model can suggest where instructions sound underspecified, but a simulated newcomer is still a simulation. If you never show the model which screens a new assistant can access, a convincing rewrite is no evidence that those screens exist or that a button is available. Use AI to find questions for the guide, and check the answers in the actual workflow.

Make the starting state and the stopping point visible

Return to the studio example. Instead of saying ‘open the board’, name the board and say how the reader reaches it. For a batch with a similar name, explain how to distinguish it: match the dispatch date and collection location to the order record. If the new assistant cannot release orders, give them the step they can perform: mark the batch ready for a supervisor and record where the request went.

Add an observable sign that the step worked. In a fictional order system, a supervisor’s release might change the order’s status to ‘Released’ and record the time. A request for release might say ‘Awaiting review’. Those states should be described separately. Otherwise a reader may conclude that they have finished the job when they have merely handed it on.

Avoid solving the problem by documenting every option. A newcomer needs the facts that distinguish the right choice from plausible alternatives. The dispatch date matters when batch names repeat; a history of why the studio renamed its boards probably belongs elsewhere. This is the same distinction our article on useful details in AI explanations makes for learning materials: keep the detail that helps the reader do the task.

Run one walkthrough without rescuing the guide

Give a colleague who has not done the task a test record and the same level of access a newcomer would have. Start from the real entry point, such as a welcome email, rather than leaving the correct screen open. Ask the person to say what they think the next step is. Note the first place where they have to guess, ask for permission or choose between two labels. If you intervene, record exactly what you said; that is a clue to what the guide left out.

• Check the first reference. If the guide names ‘the board’, ask which one the person picked and what made that choice plausible.
• Check the permissions. If a control is missing, establish whether the person needs access or whether the task belongs to someone else.
• Check a near miss. Give the tester two batches with similar names. The guide should explain how to choose without depending on the owner’s memory.
• Check the finish. Ask how the person knows whether the order was released, handed over or left unresolved.

Use the notes to ask AI for a targeted revision. Supply the verified labels and decisions, without unnecessary customer information. Tell it to mark unknowns as questions for the owner instead of inventing a screen or step. Then repeat the part of the walkthrough that failed. If the test exposed an access problem, fix the role or the process as appropriate; editing the sentence alone cannot give someone permission.

A good onboarding guide lets a person make the next decision without borrowing the writer’s knowledge. When your tester can get to the right place, choose between lookalike options and tell whether the handover succeeded, the document has done more than sound clear. It has carried the information the reader needed.

Sources

• Camerer, Loewenstein and Weber, The Curse of Knowledge in Economic Settings, Journal of Political Economy, October 1989.
• Keysar, Barr, Balin and Brauner, Taking Perspective in Conversation, Psychological Science, January 2000.