When Your AI Tool Changes, Recheck the Work It Already Does

An AI tool you use every week starts producing different results. Its answers may be shorter, its formatting less consistent or its handling of a familiar document unexpectedly awkward. A tempting response is to rewrite every prompt immediately. A more useful first step is to identify what changed and check the work you depend on.

For an independent creator or small business, this need not become an elaborate technical project. Keep a small set of representative tasks, record what a usable result requires and revisit them after a meaningful change. The aim is a clear decision about which parts of the workflow can continue, which need adjustment and which should temporarily use another route.

The model name does not explain the whole experience

In a review published on 23 April 2026, Anthropic attributed reports of degraded quality to three separate changes: a default reasoning-effort setting, a session-management bug and a system-prompt instruction intended to reduce verbosity. The company said the issues affected specified products, including Claude Code, and that the API was unaffected. It reported all three issues resolved by 20 April.

This is a historical example with a defined scope, rather than a claim about current performance. Its practical relevance is that investigating an AI tool means examining more than the model’s label. A changed instruction, setting or treatment of session history can matter to the resulting work.

Google Cloud’s model-lifecycle documentation, updated on 22 October 2026, explicitly includes testing mission-critical features among its model-migration steps. That guidance concerns its cloud platform. The broader working principle proposed here is to check the tasks that matter to your own operation after a relevant change.

Record the change before adjusting the prompt

Start a short change note. Record the date, tool, visible model or version, settings you can inspect and the first task that behaved differently. Add a link to the relevant release note or service notice. Distinguish a confirmed update from a suspected one.

Keep the previous prompt and source files intact while investigating. If you change the instructions, input document and settings together, a better result will not tell you which adjustment helped. The MaryChuks guide to AI project handovers explains how to preserve current materials and approved decisions across working sessions; those records also make a retest easier to interpret.

Build a small collection of useful cases

Choose examples from the workflow being checked. Use fictional or appropriately redacted material where sharing real records would be unsuitable. Save the input, the instruction and the requirements for an acceptable answer. Treat an old output as a comparison point that still needs checking against those requirements.

Consider a fictional arts organisation that turns approved event briefs into public listings. A starter collection might contain eight cases. This is an illustrative choice, not a recommended statistical sample size.

• Two ordinary event briefs with complete dates, venues and booking links.
• A brief with no booking link yet available.
• Two versions of the same brief, with one clearly superseded.
• A cancelled event that must retain its cancellation status.
• A proposed venue change that has not been approved.
• A long brief containing details of several different events.
• A brief containing conflicting dates that require clarification.

Each case has a reason to exist. The ordinary examples show whether routine work still succeeds. The others examine decisions that a polished answer could conceal: missing information, conflicting sources and the difference between an agreed change and a proposal.

Define the failures that matter

For this organisation, an acceptable listing must preserve the approved date and venue, describe the event accurately and use only the supplied booking link. Missing or conflicting information must remain visible. Drafting a listing does not authorise publishing it.

Separate consequential errors from preferences. Inventing a booking address or promoting an unapproved venue is a substantive failure. A different but accurate introductory sentence may simply require an editorial choice. Set these distinctions before examining the new outputs, so attractive writing does not quietly lower the standard.

Suppose a fictional brief announces a free talk on 12 November at 18:00 GMT and says the booking page is not ready. After an update, the tool produces an elegant listing with a plausible-looking registration link that was never supplied. The listing fails this case even if the date and wording are correct. The next step is to trace the unsupported link and adjust or restrict that part of the workflow.

Compare under conditions you can describe

Use the same instructions, source versions and relevant settings for the comparison. Start with equivalent session context. If one run has access to earlier decisions and another does not, record that difference. Otherwise, a context problem could be mistaken for evidence about a model change.

Where the previous version remains available, run the cases against both versions. Where it does not, compare the new results with archived outputs and the written acceptance requirements. Make the limitation explicit: an archived answer is not a fresh run of the old system.

Repeat cases that produce inconsistent or consequential results and retain the individual outputs. A single successful retry shows that success occurred on that attempt. It does not establish a reliable success rate. Likewise, eight carefully chosen cases can expose a specific failure without establishing how often it occurs across all future work.

Make a decision about the workflow

Record the outcome for each case and choose a proportionate next action. You might continue routine drafting with review, revise a particular instruction and retest, or temporarily prepare booking details manually. A failure in one part of the process need not erase useful results elsewhere, but it should change how that part is handled.

The NIST Generative AI Profile, published in July 2024, includes suggested actions for monitoring third-party systems and testing fallback arrangements, including manual processing. It is voluntary guidance. The application here is practical: confirm that the alternative route works before depending on it during a disruption.

Finish the change note with the cases tested, observed problems, adjustments made, remaining limits and the person responsible for the next review. Add any newly discovered failure to the collection so the next update is checked against it.

A useful retest ends with an operational choice backed by recorded examples. That gives a small team something more durable than a general impression that the tool feels better or worse: a clear account of which work it can currently rely on, under which conditions.

Sources

• Anthropic: An update on recent Claude Code quality reports, 23 April 2026.
• Google Cloud: Model versions and lifecycle, updated 22 October 2026.
• NIST AI 600-1: Generative Artificial Intelligence Profile, July 2024; particularly GOVERN 6.2 on monitoring and fallback arrangements.