Imagine a repair business training a model to route enquiries to the right technician. Staff have labelled several hundred messages, but the model still sends some urgent jobs to the routine queue. The next proposal is to label another thousand. That could help. It could also buy a thousand more examples of easy jobs while leaving the urgent cases just as poorly understood.
Before committing the time, try a learning curve. Train the same kind of model on several amounts of labelled data and check each version on examples it did not train on. The result tells you how performance changed over the range you actually tested. It gives you a better question to ask of the next batch: what should it contain, and what would count as an improvement?
Make the comparison fair
The scikit-learn guide to learning curves shows how to compare training and validation scores as the number of training examples changes. The training score describes performance on examples used to fit the model. The validation score describes performance on examples held out from that fit. If you change the way you measure success or swap in an easier validation set halfway through, the difference no longer tells you much about the effect of adding data.
For a small trial, pick three or four training sizes that your team can manage. Keep the model type, labelling rules and evaluation procedure steady. Draw subsets in a way that suits the real task, and repeat the comparison where feasible so you can see whether the result depends on a lucky sample. If one customer has sent five replies about the same fault, keep the thread together. A near-copy on either side of the split could make a model look better at handling unfamiliar cases than it is.
If the live system will handle newer jobs, consider holding out a later period and using earlier records to train. Make sure each training size still contains enough examples of the important categories to make the comparison meaningful. Keep a final test set apart if you will use the validation results to choose a model or rewrite the labels. Otherwise the team can end up judging its choices on the same evidence it used to make them.
Read the curve as a clue
Suppose that, in an invented trial, the model catches 18 of 30 urgent requests on a fixed validation set after training on 200 examples, 22 after 400, and 23 after 800. The first increase looks useful. The last increase amounts to one case. Before forecasting a benefit from another thousand labels, look at which cases were gained or lost. With only 30 urgent requests, a small change in the count can make the reported rate move noticeably.
If training performance is strong and validation performance improves as you add examples, another batch of similar data may be worth testing. If both scores are poor and barely move, inspect the task before buying more of the same. The model might lack a field that helps distinguish urgency, reviewers might apply different definitions, or the selected model might be too limited. The shape of the curve suggests where to look. It does not identify the cause on its own.
In a December 2017 preprint, Joel Hestness and colleagues measured how generalisation error changed as training sets grew across several deep learning tasks. They observed regular scaling patterns in the settings they studied. That supports the value of measuring the relationship rather than guessing it. It does not establish that a repair business can extend a short curve indefinitely, especially when the next batch comes from different customers or uses different labels.
Spend the next batch on a question
If urgent jobs still slip through, read a sample of the misses with the person who schedules the technicians. A customer may use ordinary language to describe a problem the team considers urgent. Another message may be impossible to classify without knowing whether the equipment has stopped working. The first case suggests collecting more examples of how customers describe the fault. The second suggests asking for the missing information rather than expecting the model to guess it.
Track the error that changes what staff do. The repair business might care about the share of urgent jobs caught and the number of routine jobs unnecessarily escalated. Overall accuracy can hide a trade-off between them. Our guide to evaluating errors in ordered AI predictions explains why a mistake near the decision boundary can matter more than an error that leaves the next action unchanged.
- Agree which mistake the next batch should reduce and what staff would do differently if it did.
- Keep the ordinary cases in the evaluation as well as the difficult ones, so an improvement on urgent jobs does not hide a new problem elsewhere.
- Record both the score and the cost of labelling. Compare another batch of similar examples with a smaller batch chosen to cover a specific gap.
- Decide what evidence would make you stop collecting and change the task, the inputs or the review process instead.
The useful outcome of a learning curve is a decision about what to try next. If the last batch helped, find out where. If it barely moved the result, check whether your team is collecting more of the information the model already has. Both findings can save work before you ask people to label another thousand records.
