Which copy of a duplicate line do I get?

The first one. The tool walks the list from the top and keeps the first occurrence of every line, so the order of what survives is the order of your original list - unless you ask for it to be sorted.

Why are my duplicates not being removed?

Almost always a hidden difference. A trailing space, a non-breaking space, or a capital letter on one copy and not the other. Turn on 'trim lines' and, if the entries are really the same thing, the case-insensitive option.

What happens to blank lines?

Empty lines are dropped. A line with nothing but spaces is kept unless you turn on 'trim lines', which empties it and therefore drops it too.

Does it find near-duplicates?

No. Two lines that differ by a character are two different lines as far as this is concerned. It compares lines exactly, apart from the trimming and the case you switch on.

How-to guides

How to Remove Duplicate Lines From a List (and the Traps That Make It Fail)

Duplicate lines that will not match are almost always a trailing space or a different capital letter. Here is how to tell the difference.

Zahid SheerazOctober 22, 20264 min read

Every list grows duplicates. A mailing list gets pasted twice. A spreadsheet column ends up with the same value in three rows. A copy-paste from an email brings along a signature line every time. Removing them sounds trivial, and usually it is - but when it does not work, the reason is nearly always something invisible.

What the tool does, exactly

The Remove Duplicates tool walks your list from the top and keeps the first occurrence of every line. Everything after it goes.

That is worth stating plainly, because "remove duplicates" has two possible meanings when the duplicates are not identical:

Alice, Marketing
Alice, Sales
Alice, Marketing

The result is:

Alice, Marketing
Alice, Sales

The first Alice, Marketing survives and the third one is dropped - not because the values match, but because the whole line matches the first one it saw. If you wanted the last one instead, or you wanted them merged, this is not the tool for it.

Three settings change the behaviour, and each one fixes a specific trap.

Trap 1: the trailing space

This is the most common reason a list of obvious duplicates will not dedupe.

A list copied out of a spreadsheet, a CSV, or an email often carries invisible characters at the end of each line. So the two lines below look identical and are not:

apple
apple␠

The first is apple, the second is apple followed by a space. They are different strings, so a comparison that is doing its job correctly keeps both.

Turn on "trim lines". It strips the whitespace at the start and the end of every line before comparing, and the two become one.

You would be amazed how often this is the whole problem. A list pasted from a spreadsheet is the classic case.

Trap 2: the capital letter

Apple
apple

By default these are two different entries, and that default is correct - Apple and apple can be two different products, two different people, or two rows in a system that treats them as distinct.

But if you know they are the same thing, the case-insensitive option merges them, and the copy that survives is the first one: Apple.

Note the direction: the surviving line is the first one, with its original capitalisation. Turning the option on does not re-capitalise the list.

Trap 3: the line ending

If a list was written on Windows and pasted somewhere that expects Unix line endings, some lines can end with a carriage return that the other lines do not have. Visually identical, technically different.

This one is handled for you: the tool splits on both kinds of line ending, so a Windows list and a Unix list behave the same. But it is worth knowing, because it is the reason a list can fail to dedupe in some other tool and work perfectly in this one.

The three settings, and when to reach for each

Setting	What it changes	Reach for it when
Trim lines	Strips leading and trailing spaces before comparing	The list came from a spreadsheet, a CSV or an email
Case-insensitive	Treats `Apple` and `apple` as the same	You know the entries are the same thing regardless of case
Keep order / Sort	Whether the result stays in your order or is sorted alphabetically	See below - this one changes meaning, not layout

Sorting changes the meaning, not just the layout

This is the one to think about before you press it.

Keep order returns the surviving lines in the order they appeared. That is what you want for almost everything: a list of steps, a list of customers in the order you entered them, a list of filenames that correspond to files on disk.

Sort returns them alphabetically. That is what you want for a reference list, a mailing list you are about to sort anyway, or a set of tags.

But if the order carries meaning - a procedure, a sequence, an order of priority - sorting destroys it, and no amount of care afterwards will put it back. If you are not sure, keep the order.

What it does not do

It does not find near-duplicates. Two lines that differ by one character are two different lines. Acme Ltd and Acme Ltd. are two entries.
It does not remove a duplicate that appears twice within one line. It works line by line, so a, b, a keeps its two as.
It does not understand your data. It does not know that 123-45-6789 and 123456789 are the same number.
It does not merge. Duplicates are dropped, not combined.

A short checklist

Turn on "trim lines" first - it is the most likely fix and costs nothing.
The first copy wins. If you wanted the last one, this is not the tool.
Keep the order unless you are sure. Sorting destroys a sequence.
Empty lines are dropped; whitespace-only lines are kept until you trim them.
Turn on case-insensitivity only when you are sure - Apple and apple are often different.
Near-duplicates are not duplicates. Clean those by hand.
It runs in your browser and keeps no copy of your list.