A decision tree for safe Unicode normalization
Normalization should make text more predictable without erasing meaning. The safest process separates compatible replacements from typography choices and language-sensitive characters.
Key takeaways
- Start with the destination’s requirements, not a universal ASCII-only rule.
- Automatically replace only characters whose meaning survives the replacement.
- Flag ambiguous punctuation and shaping characters for a human decision.
Question 1: does the destination require an exact format?
Structured identifiers, URLs, filenames, product codes, command lines, and strict form fields reward predictable characters. In those destinations, accidental zero-width marks and non-standard spacing usually add risk without adding meaning.
Editorial copy is different. Curly quotes, em dashes, guillemets, and locale-specific spacing may be intentional. Preserve them when the destination supports typography and the marks improve reading rather than interfering with parsing.
- Strict machine-facing field: prefer stable spacing and compatible punctuation.
- Human-first publication: preserve intentional typography.
- Mixed workflow: optimize for the strictest system the text must pass through.
Question 2: can the replacement change meaning?
A curly apostrophe can usually become a straight apostrophe without changing the word. A zero-width space in a product code can usually be removed. An em dash, mathematical minus sign, joiner, or special unit may carry meaning that a simple hyphen or deletion does not preserve.
When meaning is stable, an automatic replacement is reasonable. When context decides the meaning, highlight the character and ask for review. When a character is required for language shaping, preserve it unless a knowledgeable reviewer confirms the change.
Question 3: did the destination behave better?
Normalization is successful only when the cleaned value works in its target without introducing a new error. Paste it into the real form, renderer, search box, or import pipeline and check the behavior that motivated the cleanup.
If the result is still wrong, restore the original and narrow the change. Conservative iteration is safer than flattening every non-ASCII character and trying to reconstruct lost typography afterward.