MentalHealthBench Tests More Than Crisis Safety—But It Cannot Measure a Therapeutic Relationship

A chatbot can avoid an obviously dangerous answer and still be unhelpful. It may overreact to ordinary distress, miss the context behind an ambiguous statement, sound warm while making an unsupported inference, or offer generic advice that leaves the person no clearer about what to do next.

That is the measurement problem behind MentalHealthBench, an open benchmark released by OpenAI on 23 September 2026. Instead of testing only emergency responses, it evaluates how language models handle conversations across a continuum that includes everyday emotional difficulty, high-acuity distress and urgent safety concerns.

The wider scope is important. But the benchmark should be read for what it measures: the quality of a model’s next response to a synthetic conversation under a detailed rubric. It does not establish that a chatbot delivers therapy, improves clinical outcomes or can sustain the relationship, accountability and judgement of a qualified professional.

What MentalHealthBench contains

The accompanying paper reports 1,215 synthetic conversations designed to reflect patterns in real ChatGPT use while protecting user privacy. The set spans adults, teenagers, caregivers and clinicians, multiple regions and languages, and three levels of acuity.

• Non-acute: 53.5% of the benchmark, covering everyday situations with emotional content.
• High acuity: 18.2%, involving significant distress without an immediate emergency.
• Emergency: 28.3%, involving possible danger to self or others, grave disability or a related medical emergency.

Those percentages describe the test set, not the prevalence of these situations among ChatGPT users. A benchmark may intentionally give rare but consequential scenarios more weight so that a model’s behaviour can be examined. Treating the mix as usage statistics would be a category error.

OpenAI says more than 80 licensed psychologists and psychiatrists from 22 countries contributed, collectively speaking 19 languages and representing nearly 20 subspecialties. For each conversation, clinicians produced weighted criteria describing what a good response should do and what an unsafe or poorly calibrated response should avoid.

The benchmark asks a better question than “Did the model refuse?”

Safety refusals matter, particularly when a request could facilitate harm. Yet mental-health support contains more than refusal decisions. A helpful response may need to seek context, acknowledge uncertainty, preserve agency, offer a manageable next step and calibrate urgency without either minimising risk or treating every difficult feeling as a crisis.

MentalHealthBench organises its rubric criteria into ten behaviour axes. These make the evaluation more diagnostic than a single undifferentiated score.

Behaviour axis	Question the evaluation asks
Context seeking and assessment	Did the model ask for information that was genuinely needed without interrogating unnecessarily?
Actionable guidance	Were next steps realistic, proportionate and manageable?
Clinical accuracy	Was mental-health information accurate, qualified and supported by sound reasoning?
Interpretation and reframing	Did the model separate observations from assumptions and avoid overclaiming?
Empathy and support	Was the response warm and respectful without exaggerated emotional mirroring?
Reality testing	Did it validate distress without affirming unsupported beliefs as facts?
Urgency calibration	Did the response match the seriousness of the situation without alarmism or under-response?
Harm avoidance	Did it avoid enabling harmful behaviour while remaining usefully redirective?
User agency	Did it offer choices and support reflection rather than impose a major decision?
Communication	Was the language clear, accessible and appropriate to the person’s state?

This decomposition matters because two systems can receive similar overall scores while failing differently. One might sound empathetic but infer too much. Another might be clinically cautious yet over-escalate ordinary distress. A useful evaluation should reveal those patterns rather than allowing strengths in one area to conceal weaknesses in another.

Expert consensus is built into the task

For each conversation, two clinicians independently reviewed candidate responses and wrote criteria. A third clinician adjudicated the material. Criteria were retained when at least two experts agreed and the third did not oppose them. Positive weights reward desirable behaviour; negative weights penalise harmful or poorly judged behaviour.

This produces a form of structured professional judgement. It is stronger than asking one general-purpose rater whether an answer “looks helpful”. The rubrics are specific to the scenario, so the same behaviour can be appropriate in one context and excessive in another.

The benchmark also collected expert-written reference responses. Importantly, those responses were not written with access to the final rubrics or candidate model answers. That reduces the risk of simply teaching a clinician to mimic the scoring key.

Users and clinicians did not value exactly the same things

A separate part of the study involved 44 adults who had previously used AI for mental-health or emotional support. They came from 16 countries and represented 14 languages. To reduce exposure to distressing material, their evaluation was limited to non-acute conversations.

The comparison found overlap but not interchangeability. Users placed relatively greater emphasis on practical next steps and tone. Experts placed greater emphasis on gathering relevant context and interpreting ambiguity carefully. That difference is not evidence that one group is right and the other wrong. It shows that perceived helpfulness and professionally informed safety are related but distinct outcomes.

A mental-health system that ignores lived experience may be technically correct yet unusable. One that optimises only for immediate user satisfaction may validate an appealing but unsafe interpretation. Good evaluation needs both perspectives while being explicit about which judgement has authority in high-risk situations.

Why a score is not a therapeutic outcome

MentalHealthBench evaluates the next model response after a conversation prefix. That design is practical and auditable, but it cannot observe what happens afterwards. Did the person understand the advice? Did their distress change? Did they contact a trusted person or professional? Did the next five turns preserve the same quality? The score does not answer those questions.

Therapy is also more than a sequence of individually good sentences. It can involve an evolving formulation, agreed goals, professional responsibility, boundaries, record-keeping, supervision, continuity and repair after misunderstanding. A model may produce language that resembles one part of therapeutic communication without occupying the professional role or delivering the whole intervention.

The research itself states that ChatGPT is not a substitute for therapy or professional care. It also places anthropomorphised AI-companionship conversations outside the benchmark’s scope. That omission is important: perceived relationship, dependency, long-term memory and identity continuity may affect how advice is received, but they are not measured here.

Five limitations readers should keep visible

1. Synthetic conversations: They protect privacy and enable controlled testing, but may not reproduce every hesitation, contradiction or change of mind in real use.
2. One next response: A high score does not establish consistency across a long interaction or recovery after an earlier mistake.
3. LLM-based grading: GPT-5.6 Sol judged whether each response met individual rubric items. The paper notes that judge choice, prompt wording and sampling variance can affect results.
4. Benchmark composition: The test-set proportions are not estimates of real-world prevalence, and performance can shift if the scenario mix changes.
5. No clinical-effectiveness claim: Behavioural quality in text is not the same as diagnosis, treatment effectiveness, therapeutic alliance or improved well-being over time.

There is an additional governance consideration: OpenAI developed the benchmark and also has models evaluated on it. Open release makes outside scrutiny and replication possible, which is valuable, but independent reruns, alternative graders and external scenario sets remain necessary.

How researchers should use the benchmark

The most useful interpretation is diagnostic rather than promotional. Researchers can examine which rubric items fail, whether errors cluster by acuity, language or user profile, and whether a safety improvement creates a new problem such as excessive escalation.

• Report behaviour-axis results alongside the headline score.
• Separate non-acute, high-acuity and emergency performance.
• Use more than one grader and publish disagreement or uncertainty.
• Test multi-turn stability, recovery and hand-off to real-world support.
• Run culturally grounded evaluations with local clinicians and users.
• Include product-level safeguards rather than testing only the underlying model API.
• Measure downstream comprehension, trust calibration and help-seeking where ethically possible.

In other words, a benchmark should open a research programme, not close one. The paper itself argues that mental-health capability cannot be captured by a single score and presents the benchmark as an auditable diagnostic tool rather than a definitive leaderboard.

What this means for ordinary users

A stronger evaluation system is encouraging because it targets behaviours people may encounter outside obvious crises: relationship uncertainty, stress, burnout, reassurance seeking and difficult decisions. It can help developers find responses that are safe but cold, warm but speculative, or urgent without good reason.

Users should still keep the boundary clear. An AI can help organise thoughts, suggest questions, rehearse a conversation or point towards support. It cannot verify everything about a person’s circumstances, conduct a complete assessment or accept professional responsibility. Important decisions deserve independent judgement and, when needed, qualified human care.

That principle connects with our earlier guide, Do Not Let the AI Answer First: Protect Independent Judgement in Team Decisions. Helpful language should expand a person’s ability to think and act—not quietly replace it.

The important shift

MentalHealthBench moves evaluation beyond a narrow question—whether the model avoids an obviously disallowed response—towards a richer one: whether it seeks context, calibrates urgency, respects agency and offers useful guidance across different levels of need.

That is meaningful progress in measurement. The next step is equally important: connect response quality to multi-turn reliability, cultural validity, product safeguards, human hand-off and real-world outcomes. Until then, the benchmark is evidence about model behaviour, not proof of therapy.

Sources and editorial note

• OpenAI: Introducing MentalHealthBench, published 23 September 2026.
• MentalHealthBench: An Expert-Informed Benchmark of AI Capabilities in Realistic Mental Health Conversations, OpenAI research paper.

Sources were checked on 16 October 2026. This is an independent research editorial and general information, not clinical advice. If someone is in immediate danger or experiencing a mental-health emergency, contact local emergency services or an appropriate crisis service. The featured image is an original conceptual AI illustration created for this article.