Ordinal AI: A Near Miss and a Serious Error Should Count Differently

Imagine an independent animation studio using AI to estimate how much revision a client request will take. The team uses four labels: no change, minor edit, substantial revision and full rebuild. Calling a substantial revision a minor edit may leave too little time in the diary. Calling a full rebuild ‘no change’ could mean nobody books an editor at all. Both are wrong predictions, but an accuracy score counts them as one error each.

This is a problem of order. The four labels form a scale, even if the difference between one pair of neighbouring labels is much larger than the difference between another. A useful evaluation keeps track of the distance and direction of errors, then asks what each error would make the team do.

Give the order a job

Peter McCullagh’s 1980 paper on regression models for ordinal data develops models that use the ordering of categories without requiring each category to have a numerical score. That distinction matters for a studio. You can say a full rebuild takes more work than a minor edit without claiming it takes exactly three times as long. If the team needs an estimate in hours, measure hours separately; labelling categories from one to four does not turn them into time estimates.

One way to model ordered labels is to break the task into threshold questions. Does the request need any change? Does it need at least a substantial revision? Does it need a full rebuild? Those questions are nested. A full rebuild necessarily counts as at least a substantial revision. In a 2020 paper introducing CORAL, Wenzhi Cao, Vahid Mirjalili and Sebastian Raschka addressed the problem of neural networks giving inconsistent answers to such thresholds. They evaluated their approach on age estimation from images. The study illustrates the modelling idea; it does not show that the same approach will improve an animation studio’s estimates.

Order is worth encoding only when it reflects the thing you want to predict. A list of request types such as sound, typography and colour is not a scale. Putting sound first in a dropdown does not make it less serious than colour. A studio might use one field for the type of change and another for the expected amount of work. That lets a small sound edit and a full sound rebuild share a type while receiving different levels.

Look for the mistakes that change the plan

Suppose two systems each correctly label 80 out of 100 hypothetical requests. One places each of its 20 errors in a neighbouring category. The other misses a full rebuild and calls it no change, while making 19 neighbouring errors. Both have 80% accuracy. For the manager allocating time, that single large underestimate may matter more than several minor disagreements.

Keep the counts of actual and predicted labels so you can see where mistakes go. A table with actual labels down the side and predictions across the top will show whether the tool tends to move work up or down the scale. Counting errors two or more steps away is a useful extra check, but counting steps does not price the mistakes. Underestimating a substantial revision as minor may cost more than overestimating a minor edit as substantial, even though both are one step off.

A separate threshold check can be more useful than a single average. If the studio reviews all requests predicted to require substantial revision or more, ask how many truly substantial jobs the rule catches and how many extra cases it sends to review. An estimate can be off by one category without changing that decision. Conversely, a one-step error at the review boundary can decide whether anyone looks at the request.

Check the labels before judging the model

Two experienced editors may interpret ‘substantial’ differently. Define each label with examples and counterexamples: replacing one caption is usually minor; changing the timing of multiple scenes may be substantial. Agree on how to handle a case that cannot yet be estimated because the source files are missing. Give it a separate status if needed. Treating missing information as ‘no change’ teaches the system a meaning the team never intended.

When testing a score used for review, check it against cases the team has labelled independently. Our guide to temperature scaling and confidence explains a related distinction: a score that better reflects a model’s track record does not automatically fix its first choice. An ordered prediction still needs both coherent scores and a test of whether the resulting decisions work.

• Agree on the meaning of each level and record where reviewers disagree.
• Check exact matches, large misses and underestimates separately. Show the number of examples at each level, especially when full rebuilds are rare.
• Mark the boundary that changes what staff do. Count both serious jobs missed and unnecessary reviews triggered.
• Compare the model with a simple baseline, such as a short checklist the team already uses. Revisit the evaluation when the kind of work coming in changes.

A model that understands order gives you a better way to describe a mistake. Your team still has to decide what that mistake costs. Build the evaluation around the decision the label controls, and a promising score will be easier to interpret before it reaches the production diary.

Sources

• Peter McCullagh, Regression Models for Ordinal Data (1980)
• Cao, Mirjalili and Raschka, Rank Consistent Ordinal Regression for Neural Networks (2020)