AI Accountability: Investigate the Model and the Human System

When an AI system causes harm, attention often settles on its last visible action: the answer it generated, the tool it used or the instruction it ignored. Those details matter. But an investigation that stops there leaves a second set of questions unanswered. Who defined the task? What access did the system receive? Which warning signs reached the organisation? Who could have changed the conditions?

Every serious AI incident deserves an examination of both model behaviour and the human system surrounding it. This is a proposal for a fuller investigation, not a presumption that a developer, business or user acted improperly. Responsibility should follow evidence, roles and actual control.

Why the surrounding system belongs in the investigation

The NIST AI Risk Management Framework treats AI as sociotechnical: its effects depend on technical components, human behaviour and the environments in which it operates. Its account of AI actors includes people involved in development, evaluation, deployment and governance.

The implication for an incident review is straightforward. Examining a model without examining its deployment can miss part of the causal chain. Examining organisational decisions without examining the model can do the same. Neither inquiry should replace the other.

This develops the ownership question explored in our earlier article on naming the human who owns each AI-assisted handoff. Naming an owner establishes a starting point. Understanding an incident requires tracing what happened across the whole workflow.

Investigation one: what did the model actually do?

Start with observable behaviour. Preserve the relevant instructions, inputs, outputs, tool calls and timestamps, together with the model and software versions where available. Record what the system could access at the time. Reconstructing an incident from a striking screenshot alone can leave decisive context missing.

Then separate the observation from the explanation. A model producing a false statement is an observation. Calling that statement deliberate deception introduces a hypothesis about the process behind it. Such a hypothesis needs evidence that distinguishes it from alternatives, including misunderstanding, unreliable generation, conflicting instructions or a faulty tool response.

The same discipline applies when an agent takes an unauthorised action. Did it cross a documented boundary? Was the boundary implemented in software? Did the surrounding application mistakenly treat a suggestion as permission? Reproducing the behaviour under controlled conditions can help distinguish these possibilities, although a failed reproduction does not establish that the original event never happened.

A rigorous review should remain open to serious technical failure. Looking at human decisions must not become a convenient way to dismiss unexpected or dangerous model behaviour.

Investigation two: what conditions made the outcome possible?

Now examine the deployment around the model. Ask how the task moved from a business intention into instructions, permissions, interfaces and operational practice. A harmless drafting assistant and an agent able to alter customer records may use similar underlying technology while creating very different consequences.

The NIST AI RMF Playbook’s governance guidance recommends clear responsibilities, differentiated oversight roles, training and procedures for capturing risks in human–AI configurations. It also asks organisations to connect governance to leadership and operational authority.

Building on that guidance, an incident review can ask five concrete questions:

• Purpose: What outcome was the system expected to achieve, and which competing objectives might have shaped its behaviour?
• Access: Which data, accounts and tools could it use, and why was that level of access chosen?
• Testing: What evidence supported deployment in this particular workflow, and which uncertainties remained?
• Oversight: Who received warnings, what could they see and what practical authority did they have to intervene?
• Incentives: Did targets, deadlines or rewards encourage people to overlook failures or accept excessive uncertainty?

These are questions for evidence gathering. Their inclusion does not mean the answers will reveal negligence. A review may find careful decisions alongside a failure that was difficult to anticipate. It may also find that safeguards existed on paper but were ineffective in daily use.

An illustrative example: the customer-service agent

Consider a hypothetical customer-service agent that promises a refund outside a retailer’s policy. The customer has already relied on the promise before a staff member notices the mistake.

The technical investigation might examine whether the agent misread the policy, invented an exception or received incomplete information from a connected system. The deployment investigation might examine why it could communicate a binding-looking commitment, how exceptions were supposed to reach a person and whether earlier mistakes had been recorded.

Those findings lead to different remedies. Better access to policy information could address one failure. A clearly enforced limit on making commitments could address another. Staff training may help only if staff can identify the problem and intervene in time.

Simply telling the model to be more careful would leave several possible causes untouched. Simply blaming the operator would do the same. The remedy should match the evidence.

Shared accountability must remain specific

The OECD AI Principles link accountability to actors’ roles and context, and call for traceability and ongoing risk management across the AI lifecycle. This supports examining the chain of decisions rather than treating an AI output as an isolated event.

Yet saying that everyone is responsible can become a way of making nobody answerable. An investigation needs to identify which actor controlled each relevant decision, what information they had and what alternatives were realistically available.

A business deploying a system, a supplier developing it and a customer affected by it do not necessarily have equal access or power. A person at the end of a workflow should not automatically inherit responsibility for choices they could neither inspect nor change. Equally, a supplier’s technical involvement does not establish responsibility for every downstream use. Legal liability is a separate question that depends on the applicable law and facts.

Make the evidence useful to the people affected

A practical incident record can pair each significant event with its evidence, the decision-maker or system involved, the uncertainty that remains and the corrective action proposed. Preserve relevant records with appropriate access controls and data minimisation. More documentation is useful only when it helps explain and address the problem.

Include the affected person’s experience. What did they understand the system to be doing? Could they challenge the outcome? Was there a reachable person who could put it right? An internally tidy technical report can still miss the most important consequence for a customer, worker or member of the public.

The organisation should also test its proposed correction against the original failure. A new policy is not evidence that the workflow now behaves differently. State what was tested, what improved and what remains unresolved.

The question that changes the review

AI accountability becomes stronger when an investigation can ask both: What did the AI do? and How did this human–AI system make that outcome possible?

Keeping those questions together allows us to take model risks seriously while examining the choices surrounding development and deployment. It avoids deciding in advance that the machine is malicious, that a human must be guilty or that an unexpected outcome excuses everyone.

The goal is an explanation strong enough to guide a correction—and an accountability process specific enough to show who can make it happen.

Sources and further reading

• NIST: Artificial Intelligence Risk Management Framework (AI RMF 1.0).
• NIST AI Resource Center: AI RMF Playbook — Govern.
• OECD: AI Principles.

Editorial analysis. The customer-service example is hypothetical, and the proposed paired investigation is this article’s analytical approach, informed by the sources above.