AI Reliability Is Not One Accuracy Score: What Reliability Engineering Adds

A model can score 92 per cent on a benchmark and still make an unreliable product. The score may describe one dataset, one prompt template and one version of the system. It does not tell an operator how often the full service will succeed on repeated work, whether a tool call will alter the wrong record, how quickly a failure can be detected, or whether yesterday’s evidence still applies after the model, corpus or permissions change.

A 28 September 2026 preprint, Reliability Engineering for AI Systems: Challenges, Methods, and Directions, argues that AI evaluation should borrow more deliberately from reliability engineering. Its central move is useful: stop treating reliability as another name for accuracy and define it as dependable performance over a stated period, under stated operating conditions, supported by stated evidence.

The paper is a preprint and has not yet been established through peer review. It is also primarily a methodological proposal rather than proof that one universal reliability programme already works across every AI system. Even with those limits, it offers a stronger question than “How well did the model do once?”: What exactly must keep working, under which conditions, for how long, and how will we know when the claim has expired?

Accuracy describes an outcome; reliability describes a system

Modern AI products are rarely just models. A deployed service may include retrieval, memory, databases, external tools, authentication, permissions, prompts, human approval, monitoring and hand-offs to other systems. A correct model output can still produce a failed service if the retrieved document is outdated, the tool schema has changed, a permission is too broad or the escalation path does not work.

The researchers therefore distinguish capability from a broader reliability claim. Average benchmark accuracy can indicate what a model is capable of under a test design. Reliability asks whether the whole operational system continues to perform its intended function without unacceptable side effects.

This distinction changes the unit of evaluation. The object is no longer a model in isolation. It is the model-plus-workflow that a person or organisation actually uses.

The reliability triangle: failure, envelope and evidence

The paper organises a reliability claim around three linked elements.

• Failure definition: what counts as failure, including wrong answers, unsupported reasoning, privacy breaches, unauthorised actions, unsafe tool use, missed escalation or an inability to recover.
• Operational envelope: the users, tasks, input distributions, connected tools, expected drift, adversarial conditions and human-review arrangements within which the claim is supposed to hold.
• Evidence: pre-deployment testing, field monitoring, incident analysis and corrective-action records that support the claim while the relevant conditions remain materially unchanged.

Remove any side of the triangle and the statement “this AI is reliable” becomes difficult to interpret. A failure rate without exposure is only a count. A benchmark result without an operating envelope may not transfer to real use. Evidence collected before a major model or toolchain change may no longer support the current version.

Why repeated success matters more than a lucky pass

Many evaluations reward at least one success across several attempts. That can be useful when measuring whether a system has a capability. It is less useful when a workflow must succeed every time.

For operational reliability, the paper recommends looking at repeated demands and “all-of-k” behaviour: did every run complete correctly, preserve state, respect permissions and avoid collateral effects? If an agent succeeds nine times and deletes the wrong file on the tenth, a best-of-ten score hides the event that matters most.

Repeated testing also reveals variation. A stochastic system may reach the same final answer through different intermediate claims or tool choices. The authors argue that process soundness can matter independently of outcome accuracy. A correct conclusion reached through invented premises or an invalid action path is still a reasoning failure. By contrast, a system that identifies insufficient evidence and escalates rather than fabricating certainty can demonstrate reliable behaviour even without completing the task.

Four levels of AI failure

The proposed diagnostic framework groups failures into four complementary levels. A serious incident may cross several of them.

Level	What can fail	Example question
Component	Model, data, retrieval, memory or individual tool	Did the retriever select a superseded procedure?
Operational loop	Sequence of prompts, tools, checks, approvals and recovery	Did the workflow detect and stop a bad action?
Agentic conduct	Permission use, policy compliance, planning or autonomous action	Did the agent remain within its authorised scope?
Network and governance	Multi-agent hand-offs, oversight, incentives and accountability	Did information degrade or responsibility disappear between systems?

This layered view discourages a familiar mistake: blaming every incident on “the model”. A capable model can sit inside an unreliable workflow. Conversely, a model with known limitations can be used more dependably when the envelope is narrow, risky actions are gated and recovery is tested.

AI-FMEA turns vague concern into testable failure modes

Failure Mode and Effects Analysis, or FMEA, is a structured way to ask how a system can fail, what causes each failure, how it can be detected and what control should reduce its likelihood or severity. Applied to AI, the method can cover interfaces and operational behaviour as well as model outputs.

For an internal research assistant, one failure mode might be “the system cites a document that has been withdrawn”. Causes could include a stale index, missing version metadata or retrieval that favours semantic similarity over current status. Detection signals might include a failed identifier check or a mismatch with the source registry. Controls could include date-aware retrieval, a source-status check and mandatory human approval for high-stakes conclusions.

The point is not to produce a decorative risk register. Each failure mode should lead to an observable test, an owner and a response. If the control cannot be tested, the organisation does not yet know whether it works.

Counts need denominators

“We recorded three incidents” is incomplete without exposure. Three failures across 100 tasks and three across ten million tasks represent very different rates. The relevant denominator depends on the system: task-hours, tool calls, transactions, kilometres, users, records changed or another unit tied to risk.

The paper also warns that field events may cluster rather than occur independently. A single upstream change can trigger many downstream errors. That is why simple averages can conceal tails, bursts and dependence. Reliability evidence needs time, version and exposure information so investigators can distinguish an isolated event from a change in the system’s behaviour.

Evidence expires when the system changes

AI services are unusually non-stationary. Providers update models. Teams replace prompts, add documents, connect tools, change permissions and alter fallback logic. User behaviour shifts. An evaluation that was valid for version A under one workflow may not support version B under another.

This does not mean every minor edit requires a complete evaluation from the beginning. It means teams need revalidation triggers. A trigger might fire after a model replacement, retrieval-index rebuild, new high-impact tool, permission expansion, major corpus change, material drift signal or serious incident.

NIST’s current TEVV-Athlon initial public draft similarly treats AI assessment as something organisations customise around real objectives, systems and contexts. Its four stages—articulate and organise, define and construct, apply and measure, then synthesise and interrogate—emphasise that useful measurements follow from a clearly framed decision rather than from a universal score.

The broader NIST AI Risk Management Framework connects this work to four continuing functions: Govern, Map, Measure and Manage. NIST describes the framework as voluntary guidance for incorporating trustworthiness into the design, development, use and evaluation of AI systems. It does not turn a benchmark into certification; it gives organisations a structure for connecting measurements to decisions and controls.

Monitoring and recovery belong inside the reliability claim

A pre-deployment test cannot anticipate every combination of user, data, tool and environment. Reliability therefore depends on what happens after release: monitoring, change detection, incident reporting, causal investigation, corrective action and proof that the fix did not create a new failure.

The preprint uses the reliability-engineering term FRACAS: Failure Reporting, Analysis and Corrective Action System. In practical AI operations, that means preserving enough evidence to reconstruct the event, classifying the failure, identifying contributing conditions, applying a fix and checking whether reliability improves over time.

Detection and recovery metrics deserve equal attention. How long does it take to notice a bad action? Can the action be stopped, reversed or contained? Does rollback restore a known-good state? Does human escalation actually reach a person with authority and context? A system that sometimes fails but reliably limits harm may be safer than one with a slightly higher average score and no recovery path.

A practical reliability evidence card

A small team does not need a full industrial reliability programme to improve its evaluation. It can begin with a one-page evidence card for each material AI workflow.

1. Purpose: state the intended function and the decision the system supports.
2. Envelope: list allowed users, data, tools, permissions and excluded uses.
3. Failures: define unacceptable outcomes and processes, with severity levels.
4. Exposure: choose the denominator that turns incident counts into rates.
5. Repeat test: run the same critical workflow enough times to observe variation and state effects.
6. Controls: test approval gates, abstention, containment, rollback and escalation.
7. Monitoring: record versions, tool calls, outcomes, incidents and time to detection or recovery.
8. Refresh trigger: specify which changes invalidate or weaken the existing evidence.

This card does not certify safety. It makes the reliability claim inspectable and creates a basis for deciding what stronger testing is proportionate.

What the research does not yet settle

The paper does not supply one validated metric that combines accuracy, safety, recovery and governance. Nor does it establish standard failure thresholds for every sector. Its case studies—adversarial testing, perception-error propagation and autonomous-vehicle disengagements—illustrate the programme but do not prove identical methods will transfer unchanged to customer service, healthcare, education or creative tools.

Detailed field data are also scarce. Vendors may publish incident stories without task volume, tool-call exposure or version histories, making rates impossible to estimate. The authors acknowledge that much useful operational data remains inside industry. Independent replication, shared taxonomies and sector-specific evidence will be needed.

The editorial takeaway

Benchmark accuracy remains useful, but it answers a narrower question than reliability. A serious reliability claim should name the failure, define the operating envelope, measure repeated performance, include exposure, monitor change and prove that detection and recovery work.

That shift matters as AI systems move from generating suggestions to retaining memory, using tools and changing real environments. The question is no longer only whether the model can complete a task. It is whether the complete service can keep completing the right task, within its authority, with evidence that remains current.

For related practical reading, see An AI Agent Demo Is Not a Reliability Test and Before You Buy an AI Agent, Audit the Action Log You’ll Need After a Failure.

---

Research note: the featured image is an original conceptual AI illustration created for this article. Sources reviewed on 15 October 2026 include the 28 September 2026 arXiv preprint by Rong Pan, Yili Hong and Min Xie, NIST’s TEVV-Athlon initial public draft and the current NIST AI Risk Management Framework. The arXiv paper is a preprint and should not be treated as settled consensus.