Two catalogue entries look identical on screen, yet an exact comparison says they are different. You ask an AI assistant to clean the text, and it proposes a broader transformation that makes more entries match. Before approving it, decide which differences the workflow should preserve.
Unicode normalisation is a precise text operation with several forms. Choosing one belongs in the specification for an AI-assisted import, alongside the rules for spelling, punctuation and record matching. A useful result explains both what changed and what the cleaned value is allowed to mean.
Why the same-looking word can compare differently
The Unicode Consortium’s normalisation FAQ explains that some text can be encoded as a precomposed character or as a sequence containing a combining mark. These representations can be canonically equivalent. Normalisation provides a consistent representation for comparison.
Consider the word Café. Its final accented letter can be represented by U+00E9, or by U+0065 followed by U+0301: an e followed by a combining acute accent. The code-point labels identify the stored characters. They let us describe the difference even when the rendered words look alike.
In a fictional workshop catalogue, two imported rows use those different encodings. The owner wants equivalent spellings to receive the same comparison key. That is a narrowly defined task. It does not yet authorise deleting either row or deciding that they describe the same workshop.
Make the requested form explicit
Unicode’s Standard Annex #15, in the revision dated 12 August 2026, distinguishes canonical forms NFC and NFD from compatibility forms NFKC and NFKD. NFC applies canonical decomposition followed by composition. Compatibility normalisation additionally removes distinctions that may matter in a particular context, including some superscript and subscript distinctions.
For our fictional catalogue’s accented-word comparison, choose NFC as the trial rule. Give the AI the two encodings and the expected equal result. Ask it to produce a separate comparison field, leaving the imported wording available for review.
That brief is more useful than an instruction to make every title consistent. It tells the assistant which operation to implement, which example should change and which source values must remain visible. If it suggests extra transformations, those become separate proposals to examine.
A small JavaScript check makes the difference visible
The ECMAScript specification for JavaScript’s normalize method accepts NFC, NFD, NFKC and NFKD, with NFC as the default. Passing the form explicitly makes the intended rule clear. For strings stored in variables called first and second, the comparison is first.normalize("NFC") === second.normalize("NFC").
For a reproducible example, assign first = "Caf\u00E9" and second = "Cafe\u0301". Here, each backslash-u escape specifies a Unicode character in the JavaScript string. Direct equality is false; comparing their NFC-normalised values is true. Both normalised values are Café. These cases were executed in JavaScript for this article.
The check answers an encoding question for these values. It does not test a real catalogue application or demonstrate how its database compares text. Before applying the rule to live records, run the approved examples through the actual import and comparison route.
A broader form can erase a wanted distinction
Now imagine the workshop also sells a measurement worksheet. Its catalogue contains the literal text m² in one field and m2 in another. The owner has not approved treating these spellings as interchangeable. Under NFC, the two tested values remain distinct. Under NFKC, both become m2.
That result follows the selected transformation; it is not evidence that one source value was a typing error. If the field describes an area unit, converting the superscript into an ordinary digit loses a distinction the owner may need. If the field is an identifier with an approved compatibility rule, the decision may differ.
The important review happens before the original is overwritten. Ask what the field represents and what the destination expects. A rule suitable for one comparison field should earn its place in another field through examples and a stated purpose.
A larger number of matches is therefore an incomplete success measure. In the fictional catalogue, the owner needs to inspect newly equal values and decide whether those matches are appropriate. Counting how many differences disappeared cannot settle that judgement.
Keep text equivalence separate from record identity
Suppose the two Café rows refer to separate workshop dates. Giving their titles the same comparison key would be consistent with the approved rule. Merging their bookings would be a different decision, requiring the dates and the workshop records.
For a trial import, retain the source row reference, original title, normalised comparison key and proposed match. Show the owner which rows share a key. Where an action depends on identity, compare the relevant catalogue identifiers as well. Treat an unresolved group as something to inspect.
This review process is an editorial workflow proposal. It is intended to keep the transformation understandable and reversible during the trial, rather than to guarantee that every possible duplicate will be found.
Test what must remain different
A compact set of checked cases gives the fictional owner a reviewable specification:
- Precomposed Café and the e-plus-combining-accent version: different before normalisation, equal after NFC.
- Café and Cafe: remain different after NFC; this tested operation preserves the accent distinction.
- Café and CAFÉ: remain different after NFC; changing letter case would require an additional rule.
- m² and m2: remain different after NFC but become equal after NFKC.
Keep the expected outcomes beside the implementation. When a proposed edit changes one, review the requirement before accepting the code. If a new source value reveals another ambiguity, add that case to the record.
MaryChuks.com’s guide to testing AI-written regular expressions offers a related discipline: write the acceptance rule and test its boundaries. For text cleaning, those boundaries include differences that must survive.
The finished handover should name the normalisation form, identify the field it applies to and explain what the comparison authorises. Then another person can tell whether the assistant implemented the agreed operation or quietly made a wider editorial decision.
Sources
Primary documentation checked on 29 October 2026. The catalogue and workflow are fictional; the stated JavaScript comparisons were executed. The FAQ does not display a publication date.
2 thoughts on “AI Text Cleaning: Choose What Normalisation Should Preserve”