tokentight strips whitespace, comments, and duplicate lines using plain rules, not a semantic compression algorithm. It doesn't summarize and doesn't understand meaning. It works great on some content and does almost nothing on other kinds. Below are the real numbers we measured, not the flattering average.
Content type
Standard
Aggressive
Hyper-Packed
Commented code
0%
31%
45%
Formatted JSON
0%
40%
40%
Repetitive logs†
97%
97%
97%
Clean prose (PDF/DOCX text)
0%
0%
1%
Multi-page PDF, repeating header/footer‡
0%
35%
35%
CSV, no duplicate rows
0%
0%
0%
CSV, with repeated rows‡‡
0%
96%
96%
Real production log (Apache access log)§
99%
99%
99%
Real production log, with stack traces (Hadoop)§
93%
93%
93%
Real README, real novel, real code/CSS/CSV/JSON¶
unchanged
unchanged
unchanged
† With Log Collapsing turned on. It's a separate toggle from the level slider, built for repeated lines like heartbeats and retries.
Where this helps
Commented, whitespace-heavy code sees a real cut: 31 to 45%. Repetitive logs do best of all. In our own test, Log Collapsing took a 1,910-token log down to 58 tokens by collapsing 60 near-identical heartbeat lines into one marker. Pretty-printed JSON also does well from Aggressive onward. We parse it to confirm it's valid, then re-serialize it minified, dropping indentation that was only ever there for human eyes. Standard leaves it alone, because that level is meant to stay readable. Multi-page PDFs are a newer find. Real reports repeat the same header, footer and page number on every page, and from Aggressive onward we strip that using each line's position on the page rather than guessing from the text. An 8-page test document saved 35%, all of it noise.
Where it does almost nothing
Clean prose, and CSV with no repeated rows. Both are already token-dense: no comments, no redundant whitespace, no repeated lines. There's nothing for a stripper to remove. A PDF-extracted report saved 0% at Standard and Aggressive, and 1% at Hyper-Packed, which came from that level closing up the blank lines between paragraphs. A clean, deduplicated CSV saved 0% at every level. CSV with repeated rows is a different story, covered below. If your file is mostly unique prose or already-deduplicated data, don't expect much. Better you know now than after you've pasted it.
Patches and calendars now survive
A sixth round of testing put a real git patch through the tool and it came out wrecked: 1,011 lines down to 226, and of 208 deleted-line markers only 4 survived. The cause is almost too neat. The patch carried a commit message reading “the /* glob that deleted 573 Makefile lines” , and that /* opened a comment that ate 750 lines of the diff. The message describing the bug set off the bug.
The mistake underneath was treating // and /* */ as universal punctuation. They're C-family syntax. In a patch, a log, or plain English they're just characters. Three defences now: those markers only count as comments inside actual code, any “comment” longer than 200 lines is put back untouched, and patches are recognised outright, since a patch is not the code it describes. Diffs and calendar exports both come back intact.
When a toggle does nothing, we now say why
Because the tool won't touch content it doesn't recognise, plenty of files come back at 0% on purpose . That looked exactly like the tool being broken. You turn a toggle on, nothing happens, and you can't tell whether it was skipped or it failed.
The “What changed” summary now leads with the reason. Paste a CSV with Near-Duplicate Detection on and it says: detected CSV/table data, so Near-Duplicate Detection was skipped (rows that look alike are usually different records) . You get told what the tool decided your file was, which features that ruled out, and why, instead of a silent zero.
Hyper-Packed no longer glues rows together
Hyper-Packed, the strongest setting, works partly by deleting line breaks and mashing everything into one dense block. That is the right idea for prose and code. For any format where the line break is what separates one record from the next , it wrecked the file. A real 5,000-row genomics file came back as one line. So did a real 974-line translation catalogue, and a real 785-row CSV.
What settled it: we measured what the gluing actually bought on those files. It was zero . Removing the line breaks saved no tokens, because the tokenizer re-splits the glued text about the same way. All damage, no saving. Hyper-Packed now leaves line breaks alone for CSV, TSV, record-per-line JSON, config files, translation catalogues and subtitles. It still trims, still drops blank lines, it just doesn't weld the rows together.
Three formats pay a little for that. A real .env drops from 29% to 18.3% at this setting, subtitles by 0.9 points, a real .ini by 0.4. We'll take readable settings over 10 points: a .env value can contain spaces, so glue 43 of them onto one line and there's no telling where one ends. Prose and code are untouched and still get joined, which is what the setting is for.
How we found it matters, because it wasn't another pile of test files. We keep a set of rules the tool has to obey for any input: never hand back more than you were given, never turn valid data into invalid data, leave protected formats byte-for-byte alone. A few thousand generated cases run against those rules in about three seconds. Extending one rule to the strongest setting reported the damage straight away, and named every case.
Where the savings went back up
Once we measured with the real tokenizer instead of an estimate, some files looked suspiciously bad. They were. We'd been leaving safe savings on the table in three places, all now fixed:
Comment styles we simply couldn’t see. A real php.ini is 95.7% comments by weight and we were reporting 0.3% on it, because ; meant nothing to us. A real SQL schema dump is 21.3% -- comments and reported 5.8%. A real LaTeX paper carries 147 % comment lines we ignored entirely. All three are supported now: .ini went 0.3% → 95.2% , SQL 5.8% → 28.4% , LaTeX 0.3% → 4.3%.
These switch on by file extension only . We never guess them from the text, for good reason: % is a modulo operator and a CSS unit, -- is the decrement operator in C and JavaScript, and ; ends a statement across that whole family. Guess wrong and you delete the back half of somebody's for (i--; ...) loop. Upload a file and we use what its name tells us. Paste the same text with no filename and we do less.
A guess overruling a fact. A # comment and a Markdown heading are the same character, so we read the text to decide which it was, even when you'd handed us a file named .py. Real Python (120 comment lines) and real R (197) read as Markdown and kept every one, reporting near 0%. The filename now wins when there is one. Python 0.1% → 18.8% , R 0% → 47.8% , and a genuine Markdown README still correctly reports 0%.
Across all 50 real test files the average went from 28.5% to 32.3% . The buggy version we corrected earlier averaged 34.6%. That gap is the part that was never real, and we're not going back for it.
We removed a feature rather than keep patching it
Smart Line Stripping is gone. It scored every line and deleted the lowest-value ones, and across five rounds of real-data testing it damaged every single document it ever acted on . It deleted section headings and a code fence from a real README, settings from a real Laravel .env, table markup from a Chinese Wikipedia article, and then the one that ended it: the entire definitions section of the GNU GPL, including “This License” refers to version 3 , a “covered work” means , and the clause saying the licence text may not be changed.
This wasn't a tuning problem. The scoring rewarded numbers, dates and technical identifiers, so a legal definition, being a plain sentence with no figures in it, always scored near the bottom. In a licence, a contract, or a set of medical instructions, the sentences without numbers are often the ones that matter most. We narrowed the feature twice trying to save it. Narrowing it to prose only left it doing damage exclusively to the documents where precision matters most.
It never earned its place either. On 50 real datasets its only measurable effect was on long articles. Near-Duplicate Detection, which does the real work on logs (90 to 99%), is untouched. If you relied on Smart Line Stripping, Aggressive and Hyper-Packed still do everything else they did before.
A correction: we published a 92% number that was partly a bug
An earlier version of this page said a real Dockerfile compressed by 92%. It doesn’t. The real figure, measured with the actual cl100k tokenizer, is 31% . Getting there took correcting two separate mistakes: one in the tool, one in how we were measuring it.
The cause: /* starts a comment in C, JavaScript and CSS, and is also an ordinary file glob in every shell script, Dockerfile and Makefile ever written. We knew that and had a guard for it, but the guard only covered a file containing one stray glob. A real Dockerfile has two (/var/lib/apt/lists/* and "$extDir"/*.so), so the first one opened a “comment” that the second one “closed”, and everything in between was deleted as the comment body. The deleted text counted as savings, which is how a corruption bug ends up looking like a compression win on a page like this one.
On git's own Makefile the same bug deleted 573 lines, 14% of a real build file, including a working install: target. Fixed: /* is now only treated as a comment when it follows whitespace or a statement delimiter, the way a real comment does, rather than the tail of a path segment. Being closed by coincidence doesn’t make something a comment. In plain characters, the old code kept 619 of that Dockerfile’s 6,091 characters; the fixed code keeps 4,075.
The second mistake was ours, not the tool’s, and it’s the more embarrassing one. Our own test harness couldn’t load the tokenizer, so it silently fell back to an estimate that divides characters by 3 if the text looks like code and by 4 if it doesn’t. That divisor gets judged separately on the input and the output, so a step that merely stops text looking like code appears to save an extra 25% while removing nothing at all. Every stress-test percentage we’d quoted came through that estimate. The browser was always using the real tokenizer, so what users saw was never affected, but what we saw was. The harness now loads the real encoder and a test fails if it ever stops.
Re-measured properly, the headline log numbers held up exactly (99.2%, 96.6%, 94.9%, 94.8%, 93.6% on five real production logs). Several others came down and are stated as measured: a real Maven pom.xml is 17%, not 48%; a real FDA JSON export 17%, not 44%; a real LaTeX paper 3.6%, not 26%. If a number on this page moved down, that’s the estimate being replaced by a real count.
CSV with repeated rows: same win, safer mechanism, lower number
This row used to read 99% at every level, and it got there the wrong way: Log Collapsing wasn’t restricted to files that look like logs, so it collapsed runs of near-identical lines in any file. That's fine when the repeated rows really are redundant, and destructive when they aren't. The same mechanism erased 4,966 of 5,000 real genomic records in a variant-call file, because stripping the numbers out of two different records makes them look identical. Log Collapsing no longer runs on CSV at all. The saving now comes from the CSV-aware duplicate-row removal instead, which compares actual parsed fields rather than lines that merely resemble each other. Measured on a real 50-row repeated CSV, that's 0% at Standard and 96% at Aggressive and Hyper-Packed. Lower than the number we used to print, and reached by a method that can't merge two rows that were never the same.
Stress-tested against 40 real, unmodified datasets, in four rounds
Every number on this page used to come from files we wrote ourselves. We've since pulled 40 real, public datasets instead, across four separate rounds of testing, specifically looking for more bugs rather than confirming what we already believed: a real Python library (requests), a real JS library (axios), a real stylesheet (normalize.css), a real GitHub Actions config, a real Titanic passenger CSV, a real open-source README, an entire real novel (Pride and Prejudice, Project Gutenberg), real Q&A data from a Hugging Face dataset, a real minified production JS bundle (jQuery, one ~87KB line), a real SQL schema dump, a real Maven pom.xml, a real Dockerfile, a real Laravel .env.example, a real Jupyter notebook, real Chinese-language Wikipedia prose, a real production php.ini, a real messy CSV with quoted commas and empty fields, structured NDJSON logs, three real production server logs from the loghub research corpus (including one full of real Java stack traces), a real US county-level COVID-19 time series (JHU/CSSE), a real FDA adverse-event drug-safety export, real Go source from Kubernetes, real Rust source from ripgrep, a real genomics VCF file of GATK variant calls, a real captured terminal session with ANSI color codes, the real LaTeX source of a well-known ML paper (arXiv), GitHub's real public GraphQL schema, a real deeply-nested OpenAPI spec, a real Arabic Wikipedia article, four more real production logs in four different formats (Linux syslog, Zookeeper, macOS, OpenSSH), a real Rust Cargo.toml, real Terraform, a real Django translation catalogue, real WebVTT subtitles, git’s own 4,000-line Makefile, and real news articles. Near-Duplicate Detection and Log Collapsing ran against every one of them, at every level (Smart Line Stripping did too, until this testing got it removed; see below). Where these techniques win safely, it's on real logs: 99%, 96.7%, 94.6%, 94.1%, 93%. The rest is where we kept finding limits, covered next, including a genomics VCF file that now correctly shows 0% at every level, for the same reason a CSV or JSON file does. It's protected, not ignored.
Two techniques we scoped down after finding out why
Smart Line Stripping and Near-Duplicate Detection both work line by line, which is exactly right for logs and exactly wrong for anything with structure. We found this the hard way. Turned loose on the real Titanic CSV, Near-Duplicate Detection folded 32 different passengers, with different ages and fares, into one row, because it erases numbers before comparing lines (correct for a log's timestamp, wrong for a CSV's actual data) and those 32 people happened to share the same sex, class, and port of embarkation. Turned loose on real Q&A data pulled from Hugging Face, it did worse: short structural lines like "}," and "answer_start": [ repeat across totally unrelated JSON objects, and collapsing them produced output that no longer parsed as JSON at all. Smart Line Stripping had its own version of the same bug: run on the real normalize.css stylesheet, both techniques together took a file with 34 balanced {/} pairs down to 34 opens and 1 close. Not smaller CSS. Broken CSS. And on a real project README, Smart Line Stripping deleted the section headings and the closing ``` of a code sample, which doesn't just lose content, it makes every line after that fence render as if it were still inside the code block.
A second round of testing found the same failure mode in a format we hadn't considered: real .env and .ini config files. Every line in one of those is a distinct setting, no more disposable than a CSV row. A real Laravel .env.example lost SESSION_DRIVER, REDIS_PASSWORD, and MAIL_PASSWORD to Smart Line Stripping, and a real production php.ini had 22 quite different directives (precision, max_execution_time, session.cookie_lifetime, and others) folded together by Near-Duplicate Detection, purely because short "key = value" lines look alike regardless of what they configure. The same round also caught a subtler bug in how the fix itself was wired up: both techniques run partway through the pipeline, after some earlier steps have already altered the text, so checking "is this safe to touch?" against that already-altered text instead of the original input could miss the danger. A real Dockerfile proved it. Once its comments were stripped, the remaining lines no longer looked enough like code to trip the guard, and a real package name and a real chown command were deleted. Fixed by deciding "is this safe?" once, from the untouched original text, before any other step gets a chance to disguise it.
So both techniques now refuse to run on anything detected as CSV, JSON, code, Markdown, or .env/.ini-style config, reusing the content-type checks the tool already had rather than new detectors invented for the occasion. On those formats you'll see exactly 0% difference with either toggle on or off, every time, on purpose. We'd rather a toggle visibly do nothing than corrupt your data behind your back. Logs and non-code prose are unaffected and get the full benefit, confirmed again in round two: a real minified JS bundle, SQL dump, XML file, Jupyter notebook, and messy CSV all came back byte-for-byte unaffected by either toggle.
One residual gap we're disclosing rather than papering over: real Chinese-language Wikipedia text still triggered a smaller version of the same problem. The raw text carries embedded MediaWiki table markup ({| ... |}), which isn't Markdown, CSV, JSON, or code by any check we have, and a single real article's 5 real tables came back with only 1 opening and 1 closing marker left. If you're pasting raw wiki markup (rare, but it happens), treat Near-Duplicate Detection and Smart Line Stripping with the same caution as any other structured format until we've built a check for it specifically.
A third round found the same blind spot in an older, unrelated feature
Log Collapsing predates both techniques above and runs first in the whole pipeline, and it turned out to have none of their protections. It collapses any run of 3+ consecutive lines that look alike once numbers are stripped out, and on a real GATK-called VCF file (genomics variant calls, tab-separated), that erased 4,966 of 5,000 real, distinct variant records: different chromosome positions and quality scores look identical once the numbers are gone. On a real FDA drug-safety export (deeply nested JSON), it collapsed 36,116 of 67,681 lines and the result no longer parsed as JSON. We checked by parsing it, not by looking at it. And on real Go and Rust source code, it repeated the exact {/}-corruption bug from round one, through a completely different code path: three real closing braces at three different nesting depths look identical once the whitespace around them is trimmed for comparison, so real code lost real closing braces.
Fixed the same way as round two. Log Collapsing now refuses to run on anything detected as CSV, JSON, code, Markdown, or .env/.ini config, plus one new detector: a tab-delimited sibling of our CSV check, since VCF's columns are separated by tabs, not commas, and slipped past every existing gate. Finding the fix also surfaced two smaller, older bugs hiding underneath: our code detector only ever sampled a document's first 200 lines, so a real Kubernetes source file with a 150-line import block up top read as "not code" even though the other 3,500 lines plainly were. It now samples 200 lines spread across the whole document. And our CSV/TSV consistency check was comparing every line's shape against the first line's shape instead of the most common shape, which fell over on the VCF's own metadata header lines before its data rows ever got a fair look. Real production logs (round one's Apache and Hadoop fixtures) still collapse correctly after all of this. None of it came at the cost of the feature working where it should.
We stopped patching and changed how safety works
Three rounds of this (find a format we break, add a check for that format, repeat) and the pattern finally showed itself. Every check we'd written was a list of things not to touch, and that kind of list fails in one direction. Whatever format nobody had thought of yet was fair game by default, and only got protected after someone found the damage. The bug count was never about how careful the checks were. It was about how many structured file formats exist, and nobody finishes counting those.
So it’s inverted now. The tool works out what your text is once , before anything has touched it, and each feature only runs on the content types it’s been explicitly cleared for. Logs and prose for line collapsing, near-duplicate detection, and the URL dictionary. Anything the tool can’t confidently identify is left completely alone rather than experimented on. That last part is the real change. An unrecognised paste is now the safest case instead of the most dangerous one.
What that costs you: some pastes now show 0% where they used to show a number, and that 0% is the feature working. What it buys: two bugs we found by re-reading the code under the new model, before any user hit them. A real 4-column CSV export came back 3 columns wide, because the URL dictionary replaced a value spanning two columns with one placeholder. And a file made entirely of metadata headers was treated as ordinary prose, and had 4,970 distinct records collapsed into one line. Neither needed a new dataset to find. Both are fixed, and the structure that let them happen is gone rather than patched.
A real genome mistaken for base64, and a heading mistaken for bold text
A seventh round pulled ten more datasets in formats we hadn't tested yet: Protobuf, Terraform, GraphQL SDL, OpenAPI YAML, a Jupyter notebook, a 600KB SQL dump, LaTeX, an R script, a FASTA genome, and a 1.8MB chess PGN file. It found three real bugs.
The worst one: a real, unwrapped FASTA genome, the standard download format from every public sequence database, came back reduced from 9,672 characters to 143. Base64/Data URI Truncator matched it as a long base64 blob and truncated it, and that transform is the one entry in transformSafety() allowed to run on every content type, so nothing upstream could stop it. Fixed as a property of base64 itself, not a fact about FASTA: past 200 characters, real base64 mixes uppercase, lowercase and digits with near certainty, while raw sequence data is single-case with no digits at all. A declared data: URI still stays exempt, since content naming its own encoding outranks a guess.
The same file also classified as prose, the one shape almost everything is allowed to touch, because a line of nucleotide letters scored higher on our plain-text check than an actual sentence from Pride and Prejudice does. Prose now has to contain real word breaks to qualify, which keeps space-free Chinese prose in scope while keeping unbroken sequence data out.
The third: our Markdown detector counted a single **bold** anywhere in a document as proof the whole file was Markdown, which turns off comment stripping for it. One bold word inside a code comment made a real 543-line R script read as Markdown, dropping its savings from 47.8% to 0%. It now requires two distinct Markdown signals, or one signal repeated past a real threshold, measured against real Markdown files so plain code doesn't get misread again.
A String Dictionary bug: it could make output bigger, not smaller
Ten more real, public datasets (a minified jQuery bundle, a real Kubernetes YAML manifest, a real Dockerfile, a 2.7MB CSV of over 53,000 rows, the real Linux kernel README, a real Apache access log, a 26MB real JSON export, a real Laravel .env.example, real heavily-commented C source from Redis, and Bootstrap's compiled CSS) turned up a real one. Turning on String Dictionary by itself took the Linux kernel's README from 6,034 bytes and 1,265 tokens to 6,138 bytes and 1,315 tokens. Bigger, not smaller, from a feature whose entire job is making things smaller.
The cause: String Dictionary replaces a repeated URL or path with a short [STR1]-style placeholder, then appends a legend mapping it back. The legend always costs roughly one full copy of the value's own length, and the first occurrence of any value is always left intact, so a value repeating exactly twice only ever earns one substitution. That one substitution's saving can never outweigh the legend line it requires, no matter how long the value is. Every qualifying URL in that README repeated exactly twice.
Fixed: it now counts real occurrences first, and only dictionary-encodes a value when the total saving across every substitution clears the legend's own cost by a real margin, not just a value's length in isolation. The same document now saves 0.6% to 7.1% depending on level, with nothing lost.
Ten more real files, and one caught exactly by the safety net
Ten more datasets after that (a Protobuf schema, a real Android app manifest, pip's own pyproject.toml, Swift's standard-library Array.swift, a Docker Compose file, GitHub's own official .gitignore, a real production log from the HDFS distributed filesystem, three.js's compiled bundle, a Japanese Wikipedia article, and Node.js's CONTRIBUTING.md) turned up zero corruption bugs, and one confirmation the verifier works. pip's real pyproject.toml has a line reading "`#{issue} <url>`_", a real TOML string value that happens to contain a literal # that isn't a comment. Comment-stripping saw it, would have deleted to end-of-line, and unbalanced the file. The verifier caught the imbalance before any of it shipped and returned the file exactly as uploaded. Cost: 0% savings on that one file. The alternative was a corrupted config nobody would have noticed until it broke.
A real Lisp file's own closing paren, deleted as a comment
Ten more real datasets, in formats this tool had never seen (Julia, Perl, Solidity, a Vue single-file component, PowerShell, an Ansible playbook, reStructuredText, CMake, and the official redis.conf) plus one real bug. Real, unmodified GNU Emacs Lisp source (seq.el) came back with every one of its comments intact and 0% saved at every level, not because nothing was tried, but because the verifier refused the transform outright: attempting it would have deleted a real closing paren, ), and unbalanced the file.
The cause: #'foo is Lisp's sharp-quote syntax, a reference to foo, not a comment. (defalias 'seq-each #'seq-do) was read as #' opening a comment that ate seq-do), including the paren that closed the whole expression. The same two characters mean something completely different in R, where #' followed by a space is a roxygen documentation comment, already correctly handled since round 7. The fix isn't a Lisp-specific rule: #' immediately followed by an identifier with no space is a token (Lisp); #' followed by a space or nothing is a marker and its comment (R). Same tokenization logic the rest of comment-detection already runs on, one more case added to it. Lisp's family of extensions (.el, .lisp, .cl, .clj, .cljs, .scm, .rkt) also joined CSS and Markdown as formats where # is never a standalone comment at all, it's always reader syntax (hex/octal/binary literals, records, circular-reference labels) that a token-level guard can't practically enumerate one case at a time.
Fixed and re-verified against the same file: 8.8% at Aggressive, 17.1% at Hyper-Packed, balanced, zero refusals. Nothing was ever lost to a user, the verifier did exactly what it's for, at the cost of the entire file's savings until the actual bug got found and fixed rather than patched around.
The other nine held up. The official redis.conf is 2,731 lines and 88.8% comment lines by design, and came back 97.8% smaller at Aggressive with all 84 real configuration directives verified present, byte-for-byte, in the output, checked by string search against every one of them, not by eyeballing the diff. A real OpenZeppelin Solidity contract saved 61.5% from its NatSpec documentation comments alone. Real, unlabeled Julia source (Julia's own version.jl) pasted with no filename reads as Markdown and saves 0%, because Julia docstrings are themselves written in real Markdown, fenced code blocks and links included, so an unlabeled paste is genuinely ambiguous here, not misdetected; the same file uploaded as .jl saves 23% at Hyper-Packed, filename correctly overriding the guess as it does everywhere else.
Running total: 100 real, public datasets across ten rounds of testing , none of them written to make the tool look good. Real source code in well over a dozen languages (Python, JavaScript, Go, Rust, Swift, Kotlin, R, C, C#, Julia, Perl, Solidity, x86 assembly, and more). Real production logs from a dozen different systems (Apache, Hadoop, HDFS, Linux syslog, Zookeeper, macOS, OpenSSH, Windows, Thunderbird). Real structured data (CSV up to 53,000 rows, JSON up to 26MB, YAML, TOML, Protobuf, GraphQL and OpenAPI schemas, XML, CSL). Real config, build and infra files (Dockerfiles, .env, .gitignore, Docker Compose, Terraform, Kubernetes manifests, nginx config, systemd units, Java properties files, Prometheus metrics, PEM certificate chains, SQL schema dumps, the official redis.conf, CMake build files, an Ansible playbook). Real documents and prose (an entire novel, the GPL-3.0 licence text, Wikipedia articles in Chinese, Arabic and Japanese, a LaTeX paper, reStructuredText documentation, news articles, subtitles, a translation catalogue). Real framework and scripting code (a Vue single-file component, PowerShell, real GNU Emacs Lisp). Plus a genomics VCF file and a FASTA sequence file, an Android app manifest, iCalendar and chess PGN files, a Jupyter notebook, and a captured terminal session, because real pastes don't stay inside tidy categories either.
Upload limit raised to 40MB, and a real memory bug found while testing it
PDF and DOCX uploads used to cap at 20MB. Raised to 40MB after testing real files at that scale: a genuine 52MB PDF (a real IRS tax guide plus a real NASA technical report, concatenated) extracted in 17 seconds, and a genuine 38MB Word document (145 repeats of the actual text of Pride and Prejudice, packed as a real .docx) extracted in 23 seconds. Both clean.
That testing found something worth knowing. DOCX is a zip file, so its text can decompress to several times the file's own size, and that 38MB document unzipped to about 110MB of text. Measuring the real pipeline against increasing text sizes found memory, not processing time, is what actually runs out: roughly 120MB of browser memory for every 1MB of text. 10MB of text costs about 1.5GB. 50MB costs about 5.9GB. The full 110MB case pushed a single browser tab past 11GB and was still climbing when the test was stopped.
It also found a real gap in an existing safeguard. The app already capped how much text its own processing pipeline would handle, but that cap only applied once processing started, not when the extracted text first landed on the page. A large document's full, uncapped text sat in memory the whole time regardless. Fixed: extracted text is now capped at the point it's first loaded, not downstream. A very large PDF or Word document, past 40 million characters of actual text, now shows "Optimizing a preview sample only" instead of risking a crashed tab.
How big is too big? (Near-Duplicate Detection specifically)
Near-Duplicate Detection compares every line against every other line's fingerprint, and that costs time in proportion to size: roughly 0.25ms per line in our testing, measured linearly from 2,000 lines up to 100,000. A real 2,000-line production log processes in about half a second. Paste something 20 to 50 times larger, tens of thousands of lines, and it can take several seconds to over twenty. It still finishes, and it runs off the main thread so the page stays responsive, but on a very large paste, expect to wait. If that's your situation, turning the level down or leaving Near-Duplicate Detection off for a first pass is a reasonable way to avoid the wait.
Cache Health has its own test coverage too
Cache Health (the Prefix Check and Hit-Rate Reader tools) is separate from the optimizer pipeline above, since prompt caching is an API/Agent-SDK concept and neither check ever modifies the text you paste. It gets tested the same way regardless: 14 unit tests cover the actual math (which volatile patterns get flagged and where, the real cache-cost multipliers behind the hit-rate calculation), and browser tests drive a real Chromium instance through opening the drawer, running both checks, switching tabs, and checking dark-mode contrast on the drawer header.
A self-review after shipping it found one real issue: the Prefix Check results echoed a matched value back onto the page using a hand-rolled escape that only handled <, not > or &, instead of reusing the same escapeHtml() helper already used elsewhere in this codebase for exactly this situation. Not exploitable as shipped, a lone unescaped > can't open an HTML tag on its own, but inconsistent, and a real gap for a future edit to trip on. Fixed to reuse the existing helper, with a regression test that pastes a real script-injection payload as a fake session id and confirms it never executes.
On the token counts themselves
We count tokens with OpenAI's cl100k_base tokenizer (GPT-3.5/GPT-4). It's a real BPE tokenizer's real output, not an estimate. Claude, Gemini, and Llama each tokenize somewhat differently, so treat these numbers as a solid reference point, not an exact count for every model.
Is this lossless? Can I undo it?
Standard, mostly: it only touches whitespace that never carried meaning. Aggressive and Hyper-Packed are lossy. Comments get deleted, not hidden. Blank lines get deleted. Log Collapsing replaces 50 near-identical lines with one marker and a count; the other 49 are gone. Near-Duplicate Detection works the same way, just scattered instead of consecutive: near-identical lines anywhere in the document get folded into one, with a count. None of it is reversible. There's no stored mapping back to what you started with, so if you need the original later, keep the original file. JSON is mostly the exception: whitespace minification goes through a real parser and re-serializer, so no key or value is ever altered. One part of the JSON path is not lossless though: exact duplicate objects in an array get removed, keeping the first and dropping the rest. Every kept record is byte-for-byte unchanged, but "this record appeared 3 times" becomes "this record appeared once," and that count is real information if you needed it. Same for CSV: duplicate rows and empty rows and columns are deleted outright, not hidden. (Near-Duplicate Detection doesn't apply to either format at all; see above.)
Does the AI actually understand the result?
We've verified the token counts are real. We have not run the optimized output through GPT-4, Claude, or Gemini and measured whether answer quality holds up against the original, and we won't claim otherwise. Published research on the question is mixed but leans reassuring: one study found models lean heavily on variable names (which this tool never touches) but showed no significant comprehension effect from comments being present or absent; a long-context benchmark found removing comments improved results for most models tested, though not for Gemini specifically. A separate study found comment stripping measurably hurt a weaker model on a security-review task. Take that as a reasonable signal, not a guarantee.
What this tool doesn't cover
tokentight shrinks what you send an AI, before it ever reaches a model. It has no way to shrink what the AI sends back. That's the other half of the cost problem, and other tools are built specifically for it: coding-agent skills that instruct the model itself to reply more tersely, cutting the tokens it generates on the way out. Nothing here conflicts with that. Use both, and you're covering input and output instead of just one side of it.
If you use an agent or chat platform that supports a persistent instruction, this is a plain one you can paste in. It won't shorten code, commands, or exact numbers, only the filler around them.
Answer tersely. Cut greetings, restating the question, and filler like "I'd be happy to help" or "great question." Use sentence fragments where the meaning stays clear. Never shorten code, commands, numbers, file paths, or exact error text. Give the full explanation when one is genuinely needed. Brevity is the default, not a hard limit.
Copy prompt