What this tool does
Duplicate lines creep into text more often than you would expect: exported contact lists, keyword research, email lists, log files, product SKUs, URLs copied from several pages and notes merged from different documents. This tool reads your text line by line, remembers each line it has already seen, and drops any later repeat. The first occurrence of every line is kept in its original position unless you choose to sort. It then reports how many lines were read, kept and removed, so you can see at a glance how much clean-up it did.
The options
- Case-sensitive: when on, "Apple" and "apple" are different lines. Turn it off to treat them as the same, which is usually what you want for email addresses and keywords. The first version found is the one kept.
- Trim spaces: removes spaces and tabs at the ends of each line before comparing. Without this, "item" and "item " (with a trailing space) look different even though they appear identical on screen, which is a very common cause of "duplicates that will not go away".
- Remove empty lines: deletes blank lines completely. If you turn it off, the first blank line is kept and later blank lines count as duplicates and are removed.
- Order: keep the original sequence, or sort alphabetically A to Z or Z to A. Sorting uses your browser's locale-aware comparison, so accented letters sort sensibly.
Common uses
- Cleaning email or subscriber lists before importing them anywhere.
- Deduplicating keyword lists from several research sources.
- Producing a unique list of URLs, domains, IDs or tags.
- Tidying a to-do list or reading list assembled from multiple notes.
- Removing repeated entries in code, configuration or log output.
Tips for reliable results
Exact matching is deliberate. Two lines are duplicates only if they are identical after the options you pick are applied, so "http://example.com" and "http://example.com/" stay separate, and so do names with different spellings. Normalise your data first if you want those merged, for example by lowercasing and trimming. If the result looks wrong, check for invisible differences such as trailing spaces, non-breaking spaces or different line endings. This tool understands Windows (CRLF) and Unix (LF) line endings.
All processing happens in your browser, so lists containing customer details or other private data are not uploaded anywhere. Very large lists of many thousands of lines work well on modern devices, because the tool uses a set lookup rather than comparing every line to every other line.