2026-07-02 · Calc-Tools Editor

HTML Stripper: Remove HTML Tags, Get Clean Text

Almost every pipeline that consumes web text starts by stripping HTML. It looks like a one-line job — delete the angle brackets — but it is really a dimensionality reduction: a document with structure, scripts and entity references collapses into a single stream of characters. Information is always lost. The only real question is which information you choose to lose, and who downstream has to live without it.

This guide works backwards from the consumer of the text to pick a stripping depth, then walks through a reproducible sample that compares three strategies side by side, and finishes with the ordering trap and the compliance boundary that cause most batches to be redone. If you only need one fragment converted right now, the HTML stripper tool is enough; the problem this article solves is discovering, after an entire corpus is built, that the output is unusable.

1. Four Downstream Consumers, Four Hard Constraints

A stripping strategy should not be chosen by feel. It is dictated by what the output has to support.

• NLP corpus building. The model sees term frequencies and sequences. A single var t=Date.now() pushes var and function into the high-frequency list, and once noise enters the statistics, tokenisation, vectorisation and keyword extraction are all biased.
• Translation and localisation. Translation memories align by segment. If paragraph boundaries collapse into one line, the sentence breaks between <p> elements are gone and translated segments get reinserted in the wrong place.
• Search and content analytics. Index size and token quality both suffer from noise. Leave navigation, footers and cookie banners in place and "Login", "Sign up" and "Home" become the most frequent terms on your site.
• Archival and migration. The output has to be human-readable, diffable and durable. A plain-text archive that loses heading levels can only be rebuilt by re-crawling the source years later.

All four constraints reduce to one rule: choose the stripping depth by working backwards from the consumer, never from the tool's default output.

2. Three Stripping Depths

The three depths differ not in how clean they are — all of them remove tags — but in the direction of the loss. Plain text discards structure; Markdown discards rendering detail. The trap is a mismatch in either direction. Link targets cannot be recovered from a plain-text output, because the URL only ever existed inside an attribute and disappeared at the moment of stripping. Conversely, if the consumer only counts terms, the extra parsing cost of preserving structure buys nothing.

A useful ordering question: will anything render this text again? If yes, choose Markdown or structure-preserving output. If no — statistics, indexing, warehousing — plain text is enough, provided you sanitise first.

3. Real Failure Modes, Their Causes and How to Locate Them

The six problems below recur in batch corpora, and they share one trait: a single page looks perfectly fine, and the defect only appears after the whole batch runs.

The most valuable column here is not the fix but the detection method: almost all of these surface through statistical signals rather than through reading the text. In a 100,000-page corpus, manual sampling covers the first few hundred pages; noise-term hit rate, average line length and whitespace distribution can be scanned across every page.

4. Worked Example: One Fragment, Three Outputs

The sample below is a policy block from a product page. It is 440 characters long, of which script and style account for 97 characters, or 22.0% of the raw size:

<div class="post">
  <h2>Refund policy</h2>
  <script>var t=Date.now();track('view',t);</script>
  <style>.note{color:#888;font-size:12px}</style>
  <p>Refunds within 7 days&nbsp;&mdash;&nbsp;keep the packaging.</p>
  <table>
    <tr><th>Category</th><th>Window</th></tr>
    <tr><td>Digital goods</td><td>15 days</td></tr>
    <tr><td>Food</td><td>Not eligible</td></tr>
  </table>
  <p>See <a href="/refund">refund details</a>.</p>
</div>

Strategy one: blunt regex. Delete everything matching <[^>]+>, then collapse runs of whitespace into single spaces. The result is 217 characters on one line, of which 65 characters are leftover script and style — 30.0% of the output. Entities stay undecoded, so  —  remains as literal text. Half cleaned, half contaminated.

Strategy two: sanitise, map block tags to newlines, decode entities. The output drops to 130 characters across 6 lines, entities resolve correctly to a space and an em dash, and the ratio to the raw HTML is about 29.5%. But the table exposes the ceiling of plain text: the two header cells fuse into "CategoryWindow", and the data row becomes "Digital goods15 days". Cell boundaries simply do not exist in plain text.

Strategy three: convert to Markdown. The output is 174 characters across 6 lines, a ratio of about 39.5%. The heading becomes ## Refund policy, the table becomes a pipe table, and the link survives as [refund details](/refund). The cost is that the artifact is no longer plain text, so the consumer must accept lightweight markup.

Scale those ratios up: 100,000 pages at an average of 180 KB of HTML each is roughly 17.17 GB raw, and about 5.07 GB after strategy two. Storage was never the bottleneck. The real cost is cleaning and rework — and rework is almost always caused by getting the order wrong.

5. Batch Ordering: Sanitise Before You Strip

Scale the 30.0% figure to a whole corpus and the ordering question answers itself. The pipeline that holds up is:

1. Normalise encoding to UTF-8 first, so mixed charsets cannot corrupt everything downstream.
2. Sanitise: remove script, style, noscript, comments and hidden elements as whole blocks.
3. Parse with a DOM parser rather than matching tags with regular expressions.
4. Strip or convert according to the depth chosen in section 2.
5. Normalise text: decode entities, unify line endings, collapse whitespace, handle ideographic and no-break spaces.
6. Validate: scan noise-term hit rate, average line length and leftover entities across the entire batch before loading.

Steps 2 and 4 cannot be swapped. Once stripping has run, the information about which characters came from a script is gone; removing var t=Date.now() afterwards is guesswork, because it looks exactly like an ordinary English sentence. Regex replacement is unreliable for a second reason: a > inside an attribute value, prose that discusses tag syntax, or angle brackets inside a comment will all make <[^>]+> cut in the wrong place.

6. Compliance: Feasible Is Not the Same as Permitted

The table below is directional guidance, not legal advice; check the target site's terms and your local law for any real project.

Three rules of thumb: robots.txt is a floor, not a licence; leave headroom in your request rate; and stripped text is still protected expression — republishing someone's article as plain text does not make it original simply because the tags are gone.

7. FAQ

JavaScript identifiers keep appearing in my corpus. What actually fixes it?

It is almost always an ordering bug: tags were stripped before scripts were removed, so script bodies were already treated as body text. Remove script and style elements as whole blocks first, then strip tags. Adding function, var and px to a noise blocklist and measuring the hit rate before loading catches regressions.

Where is the line when crawling public pages for a corpus?

Start with robots.txt and the site terms, keep the request rate conservative, and extract only the content region you need. Research, indexing and reasonable quotation carry lower risk; mirroring a site, republishing stripped text, or commercial use after training carries substantially higher risk and needs permission first.

How should tables and lists be handled in plain text?

Plain text has no notion of column width, so cell boundaries are lost. Accept the loss, route tables through a separate Markdown pipe-table pass, or pad to fixed width and keep the header row.

The same HTML produces different output on two runs. Why?

Three common causes: the page is client-rendered, so two crawls genuinely fetch different HTML; charset detection is unstable and the same bytes are decoded differently; or entity decoding and whitespace normalisation run at inconsistent strength. Pin the encoding, pin the parser version, and keep the rules in the pipeline to make runs reproducible.

8. Three Counter-Intuitive Traps

Trap one: decoding entities before stripping tags. It feels more thorough, but it destroys prose that discusses markup. Content written as <script> in the source decodes into a literal <script> string, and the tag-stripping pass that follows deletes it. Decode after stripping, never before.

Trap two: using Unicode normalisation to "tidy up punctuation". NFKC maps full-width commas and parentheses onto their half-width forms. That helps search normalisation but is lossy for CJK typography. Corpora that must preserve CJK punctuation should use NFC; NFKC belongs in a search key, not in the final artifact.

Trap three: assuming a small sample needs no statistics. Noise arrives in template-sized batches, so ten clean pages say nothing about the rest of the site. Scanning three metrics across the whole batch costs far less than discovering the contamination after a model has already been trained on it.

9. Four Practical Rules

• Put the order in a script, not in a document. Sanitise, parse, strip and normalise as one repeatable command so that anyone re-running it gets identical output.
• Keep the mapping from source HTML to artifact. When something looks wrong you can re-run from the source instead of patching from memory.
• Run three checks before loading. Noise-term hit rate, average line length, and leftover entities — search for an ampersand followed by letters.
• Decide the depth before you start. Anything that will be rendered again needs Markdown; statistics and indexing need plain text, and that route only works when sanitising happens first.

Stripping is only the first stage of a text pipeline; character-level cleaning and format conversion come next. If zero-width characters, a BOM or ideographic spaces survive into your output, the text cleaner removes them, and the guide to invisible-character failures explains how to track them down. When the stripped result should keep headings and lists, check the rendering with the Markdown preview, and use the word counter to size the artifact before loading it.