Duplicate Line Remover

How to remove repeated lines

1. Paste one entry per line or select a UTF-8 text file.
2. Keep the default exact matching, or explicitly ignore case or surrounding whitespace. Blank-line removal is optional.
3. Remove duplicates, review the counts and repeated entries, then copy or download the result.

Does this keep the original order?

Yes. The tool compares lines across the entire list, not just adjacent repeats, and keeps the first occurrence in place. It never sorts. For example, A, B, A on three lines becomes A, B. Blank lines removed by the optional setting are counted separately from duplicates.

What do the matching options change?

By default, case and spaces matter: Apple, apple and a line with leading spaces are different. Ignoring surrounding whitespace trims only the comparison key; the retained first line is not rewritten. Ignoring case uses JavaScript Unicode lowercasing, not full language-specific case folding. Keep exact matching for case-sensitive URLs or identifiers.

Are visually identical lines always duplicates?

No. Unicode text is not normalized. A precomposed accented letter and the same letter plus a combining mark may look identical but remain different. This protects exact text by default. Chinese text, emoji, leading zeros and HTML-looking text are kept as plain text, not interpreted.

How are blank lines counted?

An entirely empty input has zero lines. A final newline terminates a record and does not create an extra record. Repeated exact empty lines collapse to one unless blank removal is enabled. Whitespace-only lines are blank for the removal option; trimming for comparison also makes them match an empty line.

What happens to line endings?

CRLF, LF and CR are accepted as separators. Output uses your selected LF or CRLF convention. A final terminator is preserved when output remains. If the last retained record is empty, a final terminator may be added to prevent that blank record from disappearing. Download uses UTF-8 and does not add a BOM; an encoding BOM is removed when reading a UTF-8 file.

Which lists is this suitable for?

Use it for plain-text lists, keywords, URLs, IDs or repeated log lines. This is not a CSV-record deduplicator: quoted CSV fields can contain several physical lines, so use a structured-data tool for those files. Oversized inputs produce an error, never a partially cleaned list. Invalid Unicode surrogate sequences are rejected rather than changed on export.