Fdupes

FDUPES is Article description::a tool for identifying duplicate files across a set of directories. It works by scanning the specified directories for files, running on those files, then running a byte-by-byte comparison on the files. It can work in tandem with duperemove, a deduplication tool for btrfs.

Configuration
FDUPES has no configuration options other than optional command-line parameters.

Find duplicate files recursively
To find duplicate files in target directories recursively the following command could be used:

Most of the time, however, it is wise to redirect the output of the command to a file:

Creating a file is a wise and efficient idea, especially when a large amount of files are being compared. It is much easier to look through a large file list with a text editor rather than attempting to parse the list via scroll back in a terminal buffer.

Find and delete files recursively
Users are strongly cautioned to run one of the above command(s) before running one of the next commands. This is done in order to verify the output is as expected. Do the operation right the first time; the fewer mistakes the better! After output is satisfactory, the following command can be used to delete all but the first occurrence of the file. Be sure to list the directories in the order of precedence so that the correct files are preserved. I.e. to keep all the files in home directory, listing the home directory last will make it show up first in the list.

The following command uses the   and    options to delete all but the first duplicate found in a file list (created using a previous command) without prompting the user:

Unmerge
No special files need to removed. Uninstall FDUPES via: