Duplicate Line Remover, Text Sorter & Whitespace Cleaner

About Duplicate Line Remover, Text Sorter & Whitespace Cleaner

This tool bundles four line-based text operations: remove duplicate lines, sort lines, clean up whitespace and blank lines, and join or re-wrap line breaks. Each one splits the input into lines and then either drops lines, reorders them, fixes the spaces inside them, or redraws the line boundaries. All four accept Windows (CRLF), old Mac (CR) and Unix (LF) line endings and output LF.

The operations are simple, but the results can surprise you, usually because "the same line" and "alphabetical order" are defined more strictly by software than by people. Invisible characters and language-specific sorting rules account for most of it.

Deduplicating: what counts as the same line

By default, each line is trimmed at both ends, comparison is case-sensitive, and the first occurrence is kept. So "apple" and " apple" are duplicates, but "Apple" is not. With ignore case on, lines are lowercased for comparison only; the kept line appears in its original spelling.

"Keep last" does more than pick the later copy. It also orders the output by where each line last appeared. That is what you want for logs or lists where newer entries are appended at the bottom and you want only the most recent one of each.

Two lines can look identical and still differ in code. The no-break space (U+00A0) is a different character from a regular space, the zero-width space (U+200B) is not removed by trimming, and é can be one code point (U+00E9) or e plus a combining accent (U+0301). The "Ignore invisible differences" option, on by default, applies NFC normalization, treats no-break spaces as ordinary spaces and ignores zero-width spaces and BOMs, for comparison only. So "a b" and "a[NBSP]b" count as duplicates, and the output keeps the first line exactly as written. Turn the option off if those differences matter to you.

Sorting: alphabetical is not code-point order

Plain code-point sorting, which is what JavaScript's default sort and many command-line tools do, puts every uppercase letter before every lowercase one ("B" before "a") and compares digits one character at a time ("file10" before "file2"). This tool instead uses the browser's Intl.Collator, an implementation of the Unicode Collation Algorithm with language-specific rules from CLDR, and passes it the language chosen under "Sort language", which defaults to the language of the page you are on.

The default strength is "base", which ignores both case and accents when ranking. "resume" and "résumé" rank as equal, and equal items stay in their input order. That is usually what people expect from a dictionary-style sort, but it means the sort will not separate them.

Natural sort (the numeric option) compares runs of digits by value, so img2.png comes before img12.png. Without it, any list of numbered files, chapters or invoices sorts as 1, 10, 11, 2.

Collation rules differ by language even for the Latin alphabet. In Spanish, ñ is a separate letter that sorts after n. In English rules, ñ is just an n with a mark, so "ñu" lands before "nube". Swedish puts å, ä and ö after z. Set the sort language to the language of the list rather than the page: a Spanish list sorted with English rules puts "ñandú" before "nube"; with Español selected it comes after.

Whitespace: CRLF, no-break spaces and indentation

The whitespace mode by default trims trailing spaces, collapses runs of spaces inside a line to one, reduces consecutive blank lines to one, and removes blank lines at the start and end. Its definition of a space includes tabs, the no-break space (U+00A0), the ideographic space (U+3000) and the other Unicode space separators, so text copied from web pages and Word documents comes out with ordinary spaces. Zero-width spaces (U+200B) and byte order marks (U+FEFF) are removed by default; the zero-width joiner used in emoji sequences is kept.

Leading indentation, including ideographic spaces used to indent CJK paragraphs, is left alone by the collapse step so code and manuscripts keep their shape. Converting tabs to spaces follows real tab-stop behavior: a tab advances to the next multiple of the tab size (4 by default), rather than always inserting four spaces.

Line endings are the other classic problem. Windows tools traditionally write CRLF; Unix tools expect LF. A shell script saved with CRLF fails with errors like "$'\r': command not found" because bash treats the carriage return as part of the command. Passing the file through this mode converts it to LF.

Joining and re-wrapping lines

Text copied from a PDF or an old email has a hard line break at the end of every visual line. The line-break mode treats blank lines as paragraph boundaries and joins the lines inside each paragraph with a space. For Chinese or Japanese text, which does not use spaces between words, choose the empty joiner so no stray spaces appear.

Re-wrapping fills each line greedily up to the chosen width (10 to 300 columns, 80 by default, the traditional terminal and email width) and breaks at spaces. A word longer than the width stays on its own overlong line unless you enable breaking long words. Width is counted in terminal columns: Latin letters take one column, and CJK characters, Hangul and emoji take two. Chinese and Japanese text without spaces is broken between characters, with basic line-breaking rules (kinsoku) so that 。, 、, closing brackets and small kana never start a line and opening brackets never end one.

Worked examples

How to use it

1. Pick a mode from the tabs: remove duplicates, sort, clean whitespace, or line breaks.
2. Paste your text and set the options (case, natural sort, blank lines, width and so on).
3. Check the result and line counts, then copy it.

Frequently asked questions