AI CSV Imports: An Invisible Signature Can Change a Column Name

The column is clearly called reference. You can see it at the top of the spreadsheet. Yet an AI-written Python import insists that the reference field is missing. Changing the spelling in the script does nothing, and opening the file again seems to confirm that everything is correct.

One possible explanation sits before the first visible letter: a UTF-8 byte order mark, often shortened to BOM. If it is decoded as ordinary text rather than handled as a file signature, it can become part of the first heading. The visible label and the key used by the programme then differ.

Start with a small, inspectable example

Imagine a fictional print seller importing a product list with two columns, reference and title. The first data record contains P101 and Café print. The intended result is a record whose reference can be retrieved by its agreed column name, while the accented title remains intact.

The example file is UTF-8 with a BOM at its beginning. Reading it using Python’s plain utf-8 decoder and the default heading behaviour of csv.DictReader produces a first key represented as \ufeffreference. Looking up row["reference"] raises KeyError.

The escape sequence shown here exposes a character that is easy to miss in an ordinary display. It is part of the key in this failed import. The reference value P101 has not vanished; the programme is asking for a different key from the one it actually created.

Understand the two stages of reading

Python’s UTF-8 signature codec documentation describes utf-8-sig: on decoding, it skips an optional UTF-8 BOM at the start of the data. This makes the treatment of that signature explicit at the point where bytes become text.

The CSV documentation explains the next stage. Unless field names are supplied separately, DictReader takes them from the first row. A character left in that first field can therefore affect the dictionary keys used for subsequent records.

These are separate responsibilities. The text decoder handles the encoding; the CSV reader handles the tabular structure. Asking AI to rename a missing column without first inspecting those stages risks treating a decoding problem as a change in the business data.

Use the decoder that fits the agreed input

For this known UTF-8 input, a file-opening expression is open(path, encoding="utf-8-sig", newline=""). Here, path stands for the actual input file location. Pass the resulting text file to csv.DictReader within the file’s normal context manager. Python’s CSV guidance recommends newline="" when opening file objects for the reader.

In the constructed example, that decoder produces the expected headings reference and title. The reference lookup returns P101, and the title remains Café print. A second version of the same UTF-8 data without a BOM also reads correctly using utf-8-sig.

This is a targeted choice for UTF-8 input with an optional signature. It is not a universal detector for arbitrary file encodings. If a supplier provides a different encoding, establish that format rather than rotating through decoder names until an error disappears.

Inspect the representation before editing the data

A useful diagnostic request to an AI assistant is: “Show the exact parsed field names using repr(reader.fieldnames), then compare them with the required headings. Explain any extra characters before suggesting a change.” The display is evidence to inspect, not an instruction to remove every unusual character.

In this example, the failing heading list appears as ['\ufeffreference', 'title']. After the appropriate decoding, it appears as ['reference', 'title']. That comparison identifies the mismatch much more directly than a screenshot of the spreadsheet.

Keep the original file during the investigation. Record how it was exported and which decoder was used. If a later import behaves differently, those details help distinguish a changed source file from a changed script.

Do not turn a file-signature fix into blanket cleaning

The boundary matters. The documented decoder skips an optional signature at the beginning of the data. It does not promise to remove the same character wherever it appears inside a title or another field. A constructed test with U+FEFF inside a title preserved it after decoding with utf-8-sig.

Whether such a character belongs inside a particular field is a separate data-quality question. Replacing it throughout every cell would be an additional transformation. Specify that transformation deliberately and inspect its effect instead of folding it silently into the import fix.

MaryChuks.com’s guide to choosing what Unicode normalisation should preserve develops this wider principle: text handling should follow the meaning and requirements of the field. A narrow correction is easier to explain when it remains separate from other cleaning decisions.

Test the import contract, including preserved text

• UTF-8 with a BOM: the first heading should match the required name.
• UTF-8 without a BOM: the same required headings should still work.
• Accented text: the sample title Café print should remain intact.
• Unexpected headings: report the mismatch before processing the product list.
• Characters inside values: keep any proposed cleaning separate from decoding.

The two decoding routes, heading lists, reference lookup and accented title were checked in Python 3.12.14 using in-memory byte streams and the standard CSV reader. Those tests confirm the stated example behaviour. They do not establish the encoding or contents of an uninspected real export.

Before using the imported records to update a catalogue, review the required headings and a small set of expected values. If the checks fail, stop with a useful explanation that includes the parsed heading names. Quietly replacing a missing reference with an empty value would hide the import problem.

Make invisible differences explainable

For an independent creator, the lesson is practical: a field that looks right may still differ from the name the code expects. Ask AI to show the parsed representation, identify the relevant file boundary and demonstrate that the agreed values survive the correction.

The useful outcome is an importer with a clear input contract: the expected encoding, signature handling, headings and review checks. That turns an apparently mysterious missing column into a small, reproducible problem whose fix can be inspected.

Sources

• Python 3.12: UTF-8 codec with BOM signature.
• Python 3.12: CSV File Reading and Writing.

Documentation checked on 5 November 2026. The seller and product record are hypothetical; the Python results were tested as described.