Temperature Scaling: Better Confidence Scores Don’t Fix Wrong Answers

Imagine a small homewares shop using a classifier to route customer emails. The model labels a message ‘delivery enquiry’ with 95% confidence. The customer is actually asking for a replacement for a damaged item. A revised version of the system gives the same answer with 76% confidence. Is that an improvement? It depends on what the shop asked the system to improve. The label is still wrong, but a more honest score could keep the message out of a queue that sends automatic replies.

Temperature scaling is one way to adjust the probabilities a classifier reports. It is especially useful as a lesson in separating two questions: which answer does the model choose, and how much trust should someone place in the number attached to that choice? The first tells you where the model sends the email. The second can help you decide whether to let it send anything at all.

What calibration measures

Take a hypothetical batch of 100 emails for which the model reports a top confidence close to 90%. If staff judge 90 of the predicted labels correct, the observed accuracy for that group is close to the reported confidence. If they judge only 70 correct, the model was too confident for that group in this batch. This comparison says nothing definitive about a single message, and a small sample can be noisy. Repeat it across score ranges and show how many cases fall into each one.

In a 2017 study on neural network calibration, Chuan Guo and colleagues compared methods for adjusting probabilities after training. They found that temperature scaling worked well on most of the datasets they examined. Their experiments involved classification models and specific image and document tasks. A shop should treat that as a reason to test the method on its own messages, rather than as evidence that its current confidence scores are reliable.

What the temperature changes

A classifier that uses softmax starts with numerical scores for its possible labels, called logits. Temperature scaling divides those scores by a single positive number before turning them into probabilities. A temperature above one brings the probabilities closer together; one below one makes the distribution sharper. Teams normally choose the temperature using labelled calibration data kept separate from model fitting, then check the result on a separate set.

Dividing all the scores for a message by the same positive number preserves their order. If delivery was the top label before scaling, it remains the top label afterwards. The top-label accuracy on the same set of messages is therefore unchanged if you still use the first choice for every message. The decision about whether to accept a first choice can change, because that decision may depend on a confidence threshold.

In the fictional shop, suppose staff review anything below 80% confidence. The damaged-item email now goes to a person instead of receiving the automatic delivery reply. That can be a better workflow even though the classifier made the same mistake. It can also increase the volume of manual work. The team needs to know how many incorrect automatic replies it prevents and how many additional messages staff will have to handle.

Test the threshold on the work you actually receive

Start with a batch the team can label consistently. Decide, for example, whether a message about a late and damaged parcel belongs in delivery or replacements, or whether it needs a separate route. If staff disagree on the label, record why. A confidence check cannot resolve a category definition the business has left ambiguous.

• Compare like with like. Run the same evaluation messages through the original and adjusted scores. Record the first choice separately from the decision to automate or ask for review.
• Look at mistakes that matter. Check whether urgent or expensive-to-fix cases are still sent automatically. Report the number of cases in each group alongside any percentage.
• Measure the handover. Count how many cases now go to people, and check that they have time and information to make a better decision.
• Check again after a change. Evaluate scores on messages from a new product, language or channel before reusing a threshold that was fitted on older traffic.

That last check is particularly useful. In a 2019 study of uncertainty under dataset shift, Yaniv Ovadia and colleagues found that calibration after training fell short as the evaluation data moved away from the familiar conditions in their experiments. A sensible implication is to check the score against the work the model now receives. A smooth display of probabilities does not show that the underlying evidence still applies.

Give the people reviewing borderline cases a way to record the final label and why it differed. An adjustment can move a misleading number below the threshold, but the review notes can reveal a more durable fix. If the same phrase keeps sending replacement requests to delivery, add that example to the next classifier evaluation.

If the team needs several possible labels for review, that is a separate design choice. Our article on conformal prediction sets explains a different question: how often the correct answer is contained in a set under stated assumptions. A better calibrated top score does not automatically supply that guarantee.

Before using a confidence number to authorise an action, write down what happens when the number crosses the threshold. If the first choice still fails on important cases, work on the classifier or narrow its role. If the score helps staff decide when to check the answer, that is a worthwhile gain in its own right.

Sources

• Guo, Pleiss, Sun and Weinberger, On Calibration of Modern Neural Networks, ICML, August 2017.
• Ovadia and colleagues, Can You Trust Your Model’s Uncertainty?, NeurIPS, 2019.