Finding and Removing Duplicate Files from Your Digital Photo Library

I needed to sort through a library of 150,000 digital photos that was taking up 150GB of disk space. I knew most were duplicates. (Poor file management practices!)

I needed a tool that would help me find the duplicates so I could delete them.

The problem was, the file were frequently renamed–so two files that were identical often had different file names. I needed a tool that would be able to recognize duplicate digital photos even if the names were different.

I found an awesome Windows program called DoubleKiller that can do just that.

DoubleKiller allows you to find duplicate files by many methods. The method that DoubleKiller offered that worked to solve my problem (duplicate digital photos with differing filenames) was to search the folders I stored my photos in and flag all files that had the same file size AND the same CRC32 signature.

(A CRC32 signature is like a unique serial number that can be calculated from a file. It’s not really unique, but the odds of two files of the same size having the same CRC32 is about 1 in 16,000,000 so it’s a pretty good indicator that the files are identical.)

One you set up the search parameters, tell DoubleKiller what folders to search in and start to run the program, it lists all the duplicate files it finds.

You can then select and delete the duplicates you want to remove.

One suggestion: DoubleKiller has a one-click feature that allows you to select all the duplicates but the first one or all the duplicates but the last one. The result set is ordered in the same order as the folders are entered in the Folders selection section of the Options tab. So when you add folders to search in the Options tab, put the folders where you want to keep files in the order of the priority that you want to keep the files. (Wow. That does not even sound clear to me–but I hope it makes sense once you play with DoubleKiller.)

It took a long time to process 150,000 files (several hours) but I did not have to waste days–or even weeks–doing it by hand and I can be sure I did not accidentally delete every copy destroying valuable work.

And now I have an extra 150GB of disk space. Awesome!

Hurray for DoubleKiller! I wish I have found this gem a long time ago!

Now to tidy up my MP3 collection…

Oh–one more thing, there is also a Pro version of DoubleKiller that runs US$ 19.95. It’s faster and offers more automation options. I think I might have to pick up a copy!