2026-07-02 · Calc-Tools Editor

Text Cleaner: Remove Extra Spaces and Blank Lines

Most guides to cleaning text explain which checkboxes to tick: trim lines, collapse spaces, delete blank lines, convert tabs. The problems that actually keep people at their desks after hours are never the visible whitespace. They are the characters you cannot see: a config line copied from a blog that makes an editor report an illegal character, two records that look identical yet fail an equality check, a CSV that lands in the wrong columns. What these have in common is that the culprit occupies no width on screen.

This guide starts at the failure site rather than at the buttons. It covers where these characters come from, which symptom each one produces, how to locate the offending column with the fewest possible steps, and how to stop them at the source. Every code point, byte sequence and program output below was verified by running it, so each one can be reproduced.

1. Why Proofreading Cannot Catch These Characters

Text is a sequence of code points; what you see is only a rendering. Glyph rendering is only responsible for characters that have a shape. U+200B zero width space, U+FEFF byte order mark and U+00AD soft hyphen produce no strokes by design, so no font size will ever reveal them. The eye notices a missing space; it cannot notice an extra invisible character.

They arrive through three channels. Rich text copying is the first: pages insert zero width spaces so very long words can wrap, and typesetting systems use U+00A0 to keep a number attached to its unit. Copying the prose takes those markers along. Generated and forwarded text is the second: model output and chat clients that soft-wrap long messages both leave zero width characters behind. Format conversion is the third: Word's non-breaking spaces, PDF layout re-flow and Excel exports all freeze their spacing decisions into plain text.

The common thread is a mismatch of tolerances. People read text with fuzzy matching; programs read it by exact code point. Every failure in this article lives in the gap between those two readings. Once that is clear, cleaning text stops being about tidy layout and becomes about removing ambiguity.

2. Five Failure Classes, Read Backwards from the Symptom

Invisible characters do not announce themselves politely. They either break a step outright or, worse, corrupt it silently. Almost everything you will meet falls into one of five classes:

The column worth memorising is the last one: every investigation begins by looking at code points, not at text. While you stay in a what-you-see-is-what-you-get view, no amount of squinting will reveal the difference; switch to a code point or byte view and it usually surfaces within seconds. That distinction is also what separates cleaning text from pretty-printing it — one handles the control symbols that carry content, the other handles how content looks.

3. Character by Character: Code Points, Bytes and What Counts as Whitespace

The table below is the core of this article. It lists the invisible characters you are most likely to meet, gives their UTF-8 byte sequences, and answers one key question: will standard whitespace handling treat this as blank and remove it? The measured column comes from JavaScript's /\s/ and String.prototype.trim(); other languages follow the same Unicode whitespace property, so the conclusions carry over.

One counter-intuitive fact hides in this table: U+200B has SPACE in its name but does not belong to the Unicode White_Space property. That property is exactly what \s, trim() and strip() consult when deciding whether something is blank. The zero width space was designed to say "a line break is allowed here", not "occupy a blank cell", so it was excluded from whitespace at the standard level. Measured: /\s/.test('\u200B') returns false, and calling trim() on a string containing it removes nothing.

That explains the most common confusion of all — "I already trimmed the spaces, why is it still unequal?" Because the cleaning function you used and the character polluting your data do not share a definition. Whitelist-based whitespace cleaning handles only what has been defined as whitespace, and it is completely blind to the five frequent offenders U+200B, U+200C, U+200D, U+2060 and U+00AD. Those have to be deleted explicitly by code point — a useful test of whether a cleaning pipeline is actually professional.

4. Two Worked Examples

Example 1: from "the length is wrong" to column 24. A config line copied from a web page reads const apiKey = "sk-live9f2c"; to the eye, but once pasted into an editor the program reports an invalid key format. Comparing the two by eye shows nothing, so the next step is to scan by code point. The output looks like this (for legibility, the zero width character in the sample output is marked by its code point):

length: 30 | code points: 30
  index 23 (column 24) -> U+200B
non-ascii count: 1
after trim(): length is still 30, trim() removed nothing

The output is unambiguous: the line holds 30 code points, and column 24 (zero-based index 23) is a U+200B sitting between sk-live and 9f2c — a spot that looks like nothing at all. Two details matter here. First, the length and the code point count agree at 30: a zero width space lives in the Basic Multilingual Plane, so one code point occupies one UTF-16 unit. A rough check like "the length is one more than I expected" can get lucky, but a check like "the length is anomalously large" will never find it. Second, trim() has no effect on it, because Unicode does not class it as whitespace. The only reliable fix is to match U+200B by code point and remove it, rather than hoping whitespace handling will do the job.

Example 2: a BOM makes JSON parsing fail outright, while U+00A0 makes fields quietly unsplittable. The same family of invisible characters behaves very differently at file level and field level. At file level, handing a BOM-prefixed UTF-8 config to a JSON parser does not produce "one extra field" — it fails immediately. The measured output is SyntaxError: Unexpected token '', "{"retry":3}" is not valid JSON, and the position it points at looks completely empty. A BOM is file-level pollution: it appears only at the very start, but as long as the parser does not ignore it, the whole file is rejected. Opening the same content in a JSON formatter fails at the first character too, which is a fast way to confirm that a BOM is the culprit.

Field level is a different story. Say a value reads Ada Lovelace, with U+00A0 where you would expect an ordinary space. Splitting on a literal space yields one field; splitting on \s yields two (measured: split(" ") gives 1, split(/\s/) gives 2). The same data produces a different result depending only on how the split was written, and this kind of failure never raises an error — it just skews the numbers. U+200B is trickier still: it is not whitespace, so \s will not split on it either. A record holding "active" and one holding "active" followed by a zero width space survive deduplication as two rows (measured: a === b is false, set size 2), and no interface will show you why. When a "these look identical" dispute appears, first work out which comparison dimension fits the case — whitespace, line endings or the characters themselves — before deciding how to clean.

5. Four Detection Techniques and Their Blind Spots

Finding invisible characters is not a matter of looking more carefully; it is a matter of changing the representation. These four techniques cover everything from a quick check to a pipeline rule:

Scanning by code point is the technique most often written incorrectly. A common mistake is reaching for \s or [^\S\r\n] to hunt for hidden characters — as shown above, that path fails entirely on U+200B. The correct approach is to spell out the ranges: match typographic invisibles with something like [\u200B-\u200F\u202A-\u202E\u2060-\u2064\uFEFF], then handle the space-like cases with [\u00A0\u3000] separately, because the two groups are treated differently — the first is normally deleted, the second normally normalised to an ordinary space. To check that an expression really hits its target, try it case by case in a regex tester; to see a whole file's differences side by side, text diff is far more reliable than the eye.

6. Four Counter-Intuitive Misconceptions

Misconception 1: I already removed the whitespace, so the text is clean. This is the most common error of all. trim(), strip() and \s only handle what Unicode defines as whitespace, and the zero width space, zero width non-joiner, zero width joiner, word joiner and soft hyphen all sit outside that definition. "I already trimmed it" is not a substitute for one code point level check.

Misconception 2: one round of Unicode normalisation fixes everything. Normalisation helps, but its reach is routinely overestimated. Measured: normalising U+3000 and U+00A0 to NFKC folds both into an ordinary space U+0020, while U+200B and U+FEFF come through NFKC completely unchanged. The reason is that normalisation deals in equivalent forms, and zero width characters have no equivalent form to collapse into. Normalisation solves half the problem; the other half needs explicit deletion, and the two belong together rather than being alternatives.

Misconception 3: if diff highlights the whole line, the content really changed. A stray U+000D at the end of a line, or a single zero width space inside it, is enough for a comparison tool to mark the entire line as different — because, character by character, it genuinely is. When a whole line lights up and the eye sees nothing, the right reaction is not to doubt the tool but to switch immediately to a code point view. The tool is not wrong; your eyes simply lack the resolution.

Misconception 4: cleaning once at the entrance is enough. If data arrives from several upstreams — web pages, manual entry, third-party APIs, legacy exports — each one brings its own pollution. Cleaning only once, just before storage, pushes every upstream difference to the last possible moment. A more robust pattern is a light normalisation pass at every entrance, so pollution is stopped as close to its source as possible. This does not conflict with "cleaning is lossy, back up first": entrance normalisation only standardises, while the actual deletions happen at validation time.

7. Frequently Asked Questions

Why does code copied from a web page trigger an illegal character error when nothing looks wrong?

Because the error is about code points, not glyphs. To allow wrapping or typesetting, pages insert characters such as U+200B zero width space and U+00A0 no-break space, which draw no visible strokes but travel with the copied text; a program comparing code point by code point then rejects it. Print the suspect string by code point instead of zooming the font — for example with [...s].map(c => c.codePointAt(0)).

Should a BOM be removed or kept?

It depends on the consumer. Source code and configs read by script interpreters, JSON parsers, version control or build tools should be saved as UTF-8 without a BOM — a BOM-prefixed JSON file throws SyntaxError in testing. In office workflows, though, a BOM sometimes acts as an encoding label: some versions of Excel rely on it to detect UTF-8 in a CSV file, and removing it can turn the text into mojibake. Decide by consumer: strip it for programs, keep it for spreadsheet users.

Why does \s fail to clean a zero width space?

Because \s matches the Unicode White_Space property, and U+200B does not carry that property despite having SPACE in its name — its purpose is to permit a line break, not to occupy a blank cell. Measured: /\s/.test('\u200B') returns false. Cleaning it requires naming the code point explicitly, for example by matching [\u200B-\u200F\uFEFF] and deleting the matches.

How do I clean a whole file instead of pasting fragments one by one?

Write a short script that works by code point: read the file, test whether the first three bytes are EF BB BF, delete zero width characters by code point range, normalise U+00A0 and U+3000 to ordinary spaces, unify line endings, and write back as UTF-8 without a BOM. Always keep a backup of the original, because cleaning is lossy and deleted characters cannot be restored. For a small amount of text, a text cleaner is faster; for hundreds of files, a script is the only sane option.

Two strings look identical but compare unequal — what should I check first?

Compare code point sequences rather than appearances. Print both strings code point by code point: if the lengths match, look at the first position that differs; if they do not, look at what the extra character actually is. In practice the extra character is usually U+200B or U+00A0 — the first from copy and paste, the second from web typesetting. Once located, decide whether to delete it or to normalise both sides to the same representation.

8. Preventing It at the Source

• Validate a character whitelist before storing anything. Define the allowed range for each field — say "printable ASCII plus CJK plus common punctuation" — and reject or flag anything outside it. Do not use \s to police the whitelist, since it is blind to zero width characters; write the forbidden code point ranges into the rule itself.
• Pair normalisation with explicit deletion. Run NFKC normalisation first to handle full-width and no-break spaces that have equivalent forms, then delete zero width characters by code point range. Do not reverse the order — doing so lets normalisation reintroduce new whitespace forms into positions you already handled.
• Put the character check into CI or a commit hook. Add a pipeline check that files parse safely under the agreed encoding, and treat stray zero width characters as lint errors. Once this kind of problem reaches the main branch, the cost of tracking it down multiplies.
• Keep the raw copy so cleaning stays auditable. Cleaning is destructive. For data you maintain over time, keep an uncleaned original: it lets you reproduce the fault and re-run the rules after you change them. To quantify the difference before and after, a word counter quickly reports how character and word totals moved.

If the messy text came from a web page or rich text, use the right order: first exclude non-content blocks such as <script> and <style> at the HTML level, then strip the tags to get plain text, and only then clean at character level. Reverse the order and script content ends up in your corpus as if it were prose — the most common trap in HTML stripping and text extraction. Before the cleaned text goes into storage or a comparison, it is worth checking how word counts are defined, so that "looks clean" really does translate into "compares equal". Only when the character layer is free of ambiguity will search, deduplication and counting give answers you can trust.