Dataset profiler

What it does

Open a JSON Lines or CSV file, or paste some rows, and it profiles the whole dataset:

• Length: tokens and characters per row — min, median, the 90th and 99th percentiles and max — with a histogram of each.
• Outliers: how many rows fall far outside the rest, and the longest rows by number.
• Scripts: which writing system each row is mostly in — Latin, Cyrillic, Hebrew, Han and so on.
• Repeated rows: how many rows repeat an earlier one, and which.
• Fields: for each field or column, how many rows fill it, leave it out, set it to null or leave it empty.
• A random sample you can read, the same sample every time for the same file and seed.

Then download the profile as a Markdown report, or the sample as a file. A sampled row longer than 20,000 characters is shown cut short, and left out of the sample file, which would not be valid JSON or CSV with it.

The file never leaves your browser. It is read in a background thread as it streams, one row at a time, so a file of hundreds of megabytes is profiled without freezing the page and without being held in memory. A shared link holds the settings and the seed, never the rows.

Try it: support chats in JSON Lines, in five scripts, or product reviews in CSV, with a new line inside a cell.

How it reads a row

JSON Lines: one JSON object per line. Its fields are its top-level keys, and its text is every string in it, at any depth — so a chat example's text is its messages. CSV: a header row, then one row per record. Fields may be in double quotes, with "" for a quote and line breaks inside; lines may end in CRLF or LF; a byte-order mark is ignored. The CSV reader is the one the batch tools use. Numbers, booleans and keys are not text.

A line that is not JSON, or a CSV record with the wrong number of fields, is counted and listed, and the rest is still profiled. A CSV record of more than 16 million characters — which is what a quote that never closes turns the rest of a file into — is not held: it is reported once, and reading goes on after it ends. A single JSON array is not read: write it as one object per line, so a file of any size can stream.

How it counts

• Tokens: exactly with OpenAI's o200k_base or cl100k_base encodings, which run in your browser, or as an estimate at about four characters a token. An exact count is exact for the OpenAI models that use that encoding; other providers count with their own tokenizers, through their APIs. A single run of hundreds of characters with no space in it is estimated rather than counted exactly, and the page says how many rows that applied to.
• Percentiles are nearest-rank. Up to 200,000 rows they use every row; past that, a seeded sample of 200,000, and they are marked as estimates. Min, max, totals and the histograms always count every row. The histograms have four buckets per doubling, so a long tail stays readable.
• Outliers are rows outside Tukey's far-out fences: more than three times the interquartile range below the first quartile or above the third.
• Scripts are found by Unicode script, from the letters in each row. This is script detection, not language identification: Latin covers English, French and Vietnamese alike; Arabic script covers Arabic, Persian and Urdu; Han covers Chinese and much of Japanese. Digits, punctuation and emoji belong to no script.
• Repeated rows are rows that are the same — for JSON Lines, after parsing, so key order and spacing do not matter; for CSV, the same cells. A local hash of each row finds the candidates, and a row is counted as a repeat only after the earlier row is read back from your file and compared with it. A JSON Lines row with bytes that are not valid UTF-8 is compared by its exact bytes, and a CSV row with such bytes is never counted as a repeat. Up to a million distinct rows are remembered; past that, the count is marked as a lower bound.
• The sample is reservoir sampling with a seeded random number generator: every row has the same chance, and the same file with the same seed gives the same rows. The seed is in the settings and in a shared link.

What it does not know

• Languages. It tells scripts apart, not languages.
• Exact tokens for Claude, Gemini or Grok. Their tokenizers do not run in the browser.
• Near-duplicates. Only rows that are the same after parsing are repeats; a row with one word changed is not.
• Whether a dataset is fit for a purpose. It describes the data; it does not judge it, and cleans nothing. The fine-tuning dataset validator checks a training file against a provider's format.

FAQ

Is my file uploaded anywhere?

No. It is read in your browser, in a background thread, and nothing is sent. The report and the sample are written in your browser too.

Why does the same seed give the same sample?

The sample is drawn with a random number generator started from the seed, so it is random with respect to the data but repeatable: change the seed for another sample, keep it to get the same one back.

Sources

The token counts use OpenAI's encodings, held to OpenAI's own tiktoken, and the estimate is Salyro's own rule of about four characters a token. The scripts come from the Unicode blocks of each script. The profile is computed in the browser from the file alone.