Chunking visualizer

What it does

Paste a document, pick a strategy, a size and an overlap, and see the split: every chunk drawn over the text in alternating colours with its number, the overlapping text striped and underlined, each chunk's own token count, a histogram of the sizes, and the figures for the whole document. Download the chunks as JSON or JSON Lines.

Try it: a Markdown guide split by heading, Hebrew and emoji split by sentence, fixed 32-token windows with 8 overlapping, or an article split recursively.

A document converted with Document → Markdown can be sent straight here.

The strategies

Each one is defined here and written for this tool; none is presented as a particular library's splitter, and none is guaranteed to match one.

• Fixed characters — each chunk takes as many characters as fit in the size; the next starts where it shares at most the overlap with it.
• Fixed tokens — the same window, measured in tokens.
• Sentences — sentences are packed whole into chunks up to the size, and the overlap repeats whole trailing sentences. A sentence longer than the size is split at spaces, then between characters.
• Markdown headings — one chunk per section: an ATX heading (# to ######) and everything up to the next heading. A # inside a fenced code block is not a heading. Each chunk keeps its heading path, such as Guide › Install. A section longer than the size is split by the recursive rule, inside the section.
• Recursive — the text is cut at blank lines; a piece still over the size is cut at line breaks, then at sentence ends, then at spaces, then between characters. At each level, whole pieces are packed into chunks up to the size, with whole trailing pieces repeated as the overlap.

For sentences, headings and recursive, choose whether the size is measured in tokens or characters.

Packing, the rule the last three share: pieces go into a chunk while it stays within the size. When the next piece does not fit, the chunk is closed, and the next one starts with as many of its trailing pieces as fit in the overlap and still leave room for the new piece. A piece over the size is split on its own, and no overlap is carried into or out of it.

Every chunk is an exact slice of your text: nothing is trimmed or rewritten, the chunks cover the whole text in order, and taking the overlap off each gives the document back unchanged. The same text and settings always give the same chunks.

Characters are never cut

A character here is what a reader sees as one: an emoji with its skin tone or its joiners (👨‍👩‍👧‍👦 is one), a flag, a Hebrew letter with its vowel points, a letter with a combining accent. The boundaries come from your browser's Intl.Segmenter, which follows Unicode's text segmentation rules (UAX #29), and every strategy cuts only at one of them. Sentence ends come from the same rules. A single character heavier than the size — an emoji can be several tokens — becomes a chunk of its own, and the page says so.

How tokens are counted

With o200k_base or cl100k_base, OpenAI's encodings, counts are exact: the same tokenizer and the same checks as the rest of these tools, against OpenAI's own tiktoken. Every other model is estimated as characters ÷ 4, rounded up — Anthropic, Google and xAI count through their own APIs, which this page does not call. The method is written beside every figure.

For the fixed-token window and a size in tokens, the whole document is tokenized once, and each chunk takes the tokens that end inside it. The count shown for each chunk is then made on the chunk's own text, so at a chunk's edges it can come out a token or two above the size; the page says when it does.

The figures

• Tokens per chunk — the smallest, the median, the largest and the mean.
• All chunks together — every chunk's tokens added up, so overlapping text counts each time it repeats, beside the document's own count. The difference is what the overlap and the cuts add.
• The histogram groups chunks by token count.

The page draws up to 300 chunks and the first 60,000 characters, and lists up to 200 chunks in the table; the downloads hold every chunk. Offsets are in UTF-16 code units, as JavaScript's String.prototype.slice takes them. The tool reads up to 1,000,000 characters and makes up to 20,000 chunks.

What it does not do

It shows how a text would be cut. It does not make embeddings, store the chunks or search them. Nothing you paste is sent to Salyro or anywhere else, and a shared link holds the settings only — never your text.

FAQ

What size and overlap should I use?

It depends on the embedding model and what the chunks are for; there is no size that is right in general. This page lets you see what a choice does to your own document before you index it.

Why does a chunk count more tokens than the size?

Tokens are measured on the whole document, and a chunk cut out of it can tokenize slightly differently at its edges. Or the chunk is one character, such as an emoji sequence, that is heavier than the size and is never cut.

Why is the count an estimate?

You chose the estimate, or the exact tokenizer could not load or took too long in this browser. The label beside the figures says which.

Sources

Grapheme and sentence boundaries: Unicode Standard Annex #29, Unicode Text Segmentation, as implemented by the browser's Intl.Segmenter. Headings and fenced code blocks: the CommonMark specification. Exact token counts: the token layer these tools share, held to OpenAI's tiktoken; the encodings are chosen on the page, not matched to a model.