Find and Remove Duplicate Files to Free Up Hard Disk Space

Posted By Raymond In Category: Computer

Dec
17
2007
Donate

One of my year 2008 resolution is to clean up and organize all my files. I have software, movies, mp3s, documents and notes everywhere in I think at least 5 separate external hard drives. All my external hard drive enclosure looks the same and I honestly tell you I can’t identify which is which unless I connect to my computer to see what’s in there. Both my laptop and desktop hard disk is running out of disk space. When I am in these conditions, don’t expect me to have a proper backup of my stuff when all my files are in such mess. I know it’s bad… really bad. Actually having less or no duplicate files is good because you can free up your hard disk space, have proper backup, reduce backup time and reduce file searching time.

So what I am going to do is I have to start organizing my files. I’ll put all software into one hard disk, all mp3s into another hard disk and so on… Then I’ll have an extra hard disk solely for backing up my important files. But before I do all that, I expect to have a lot of same and duplicate files which is taking up a lot of unnecessary hard drive space.

I’ve found a total of 6 freeware to help me find and remove duplicate files on my computer. It is interesting because I learn the methods used by each different software to scan for duplicate files.


If you do a search in google on “remove duplicate files”, you’ll easily find a lot of software that is able to do that. I’ve found 6 of them and I will try my best to find out which is the best to use. For me, most important thing in a software is how easy it is to use and also the method being used by the software do perform a task.

1. Duplicate File Finder
FREE Duplicate File Finder
- This is the easiest software to find duplicate files and also with only a click of a button, it will remove all old duplicate files. Portable, simple graphical user interface and no configuration at all. The method being used to scan for duplicate file is raw binary comparison of the actual data inside the file. It is 100% accurate using this method. Original Files are determined by creation date, if it was created first, it is considered the original file. Took 2 minutes and 29 secs to scan 106GB (23950 files). Not able to select which folder to exclude scanning, so you have to avoid adding root of a drive to the path.
[ Download Duplicate File Finder ]

2. Heatsoft Clone Cleaner Lite
FREE Clone Cleaner Lite
- Nothing much to configure. Just add the path that you want to scan and search. By default it protects Windows, Program Files and hidden folders. Took 2 minutes 5 seconds to scan 106GB (23950 files). One thing I don’t like is it promotes the PRO(shareware) version at a few places on the software. It uses TABS to show you the scan result in same name, same size, same name and size, same content, full clones and zero size. It has Smart Mark where you can automatically mark the duplicate files to be deleted.
[ Download Heatsoft Clone Cleaner Lite ]

3. DupKiller
FREE DupKiller
- DupKiller has all types of scanning methods. Same name, same date and time, same size, same type(extension), similar name and same content. The same content is raw binary comparison which is 100% accurate. Has very good exclusion option but unfortunately very slow in binary comparison. Took 4 minutes 55 seconds to scan 0.3% of 106GB (23950 files). I also don’t like DupKiller automatically saving and loading of previous scanned results by default. You can turn it off at the options page.
[ Download DupKiller ]

4. Easy Duplicate Finder
FREE Easy Duplicate Finder
- Able to exclude system folders (windows and program files). It says that it search for true duplicates (files with same contents) which I think it meant raw binary comparison. Took 1 minute 46 seconds to scan 106GB (23950 files). This can be a really good duplicate finder BUT there is no way I can easily select all duplicates and delete them with one click. It found 5809 duplicates on my computer and I am going to have a hard time selecting them one by one.
[ Download Easy Duplicate Finder ]

5. DoubleKiller
FREE DoubleKiller
- Ability to scan for duplicates using names, sizes, dates and CRC32. All methods used here is not 100% accurate. CRC32 is not 100% because there is a very small probability of two files having the same checksum although they are not identical. If you want to use raw binary comparison, you need to PURCHASE the PRO version. I didn’t test this because it doesn’t have raw binary comparison.
[ Download DoubleKiller ]

6. CloneSpy
FREE CloneSpy
- CloneSpy is another one that doesn’t do raw binary comparison. It only compares checksums, same name and approximate size. I’d totally avoid using this to scan and delete duplicate files on my computer.
[ Download CloneSpy ]

There you have it. The best duplicate file scanner and remover is Heatsoft Clone Cleaner Lite and the easiest to use is Duplicate File Finder. I’d be using this two in future to remove all the duplicate files on my computer, laptop and external hard drives.


Related posts:
  • A FREE Hard Disk utility to Benchmark and Test your hard drive
  • Find Out What Files and Folders are Taking Up Hard Drive Spaces
  • Find any files in RapidShared and Fixdown
  • Change Windows Vista System Restore Disk Space Usage
  • Accelerate Hard Disk Drive to Speed Up File Transfer
    • Colin

      Hi Raymond, Your link to Duplicate File Finder is a link to Smiley Central , It continually loads Pop Ups, (Despite my pop up blocker being set to it’s strongest level.

      I used the “Red” link above No 1. That worked fine.

      Regards

      Colin

    • http://www.raymond.cc/ Raymond

      Duplicate File Finder link updated. Sorry for any inconveniences.

    • Mahmoud

      thanx Rymond

    • Pingback: kajun36 » Find and Remove Duplicate Files to Free Up Hard Disk Space

    • MR7

      Lol, same situation here. A lot of drives, small space left and no ideia of what is inside. Very nice tools, will try it soon…

    • MR7

      Oh, and Duplicate File Finder link is still wrong, it is pointing to Web Hosting Pro, or something like that. I google it and found this link:

      http://ukms.tucows.com/files2/DupSetup.exe

      It’s for a USA mirror, but that is more mirrors here:

      http://www.tucows.com/download.html?software_id=373411&t=2

      ;)

    • http://www.raymond.cc/ Raymond

      LOL, looks like the direct link checks for referrer. I’ve fixed it once and for all :P

    • Christina

      Thanks … I was getting fed up myself with all the duplicates on my poor overused computer ;-)

    • Merlin_Magii

      Raymond,
      for a couple of years I believed ‘Black Viper’ was the most respected Guru online …. I have to say …. you have snatched that crown, in my humble opinion.
      Thank you for a very credible and useful site. My compliments – you are required reading in my arsenal now.
      Merlin
      PS – any chance of an answer to my emailed question about U3 USB Hack please?

    • Replika

      Request article: File Compare softwares

    • CHRIS

      Raymond–long complicated story but the end result is the I have duplicate files (many many) after doing a restore of an F partition–data files. I thought I could use your duplicate file finder program but when I went to the actual copies of the original they had an earlier time: 1:25 am versus actual file 9:25 am. When I tried to open the duplicate it could not find a program to open it with. That could be a problem. I don’t see an option to choose. I have lots of duplicates but it is only showing what looks like the good file name. Please help me.

    • CHRIS

      Had to undo it on my own–I will get up with you if I need help on something else. But thanks anyway.

    • http://www.moleskinsoft.com Alexander

      I’d like my software is Clone Remover (http://www.moleskinsoft.com). It can search for duplicate mp3 files by tag and similar images. And can restore files that moved into separate folder. But Clone Remover is shareware.

    • http://www.ashisoft.com Swat

      I really like Duplicate Finder. An especially nice touch was when it told me that I was deleting all copies of a
      file. I tried a few other duplicate file finders and hated them.

      http://www.ashisoft.com

    • http://www.moleskinsoft.com David

      Duplicate Finder works buggy on Vista.

    • jack

      David stop promoting your own product!!!

      i downloaded duplicate finder today and it works great under Windows Vista.

    • Kevin

      I’ve tried a couple duplicate finders from the list. They don’t seem intuitive. I’d like to identify a master folder that all of the original or unique files will remain and check the other folders on the drive against the original, only deleting files outside of the master folder.

      It would even be nice if it would transfer any remaining unique files outside of the master folder to inside that folder.

      Perhaps I’ve overlooked some existing features.

      Duplicate File Finder crashed 49% of the way through a very long scan (hours).

      Thanks

    • Phil

      Thanks Raymond a good start.
      I think the speed in the first 3 is because they don\’t do a full binary compare first up. Certainly HCCLite makes its lists based on name, size etc. Then it will do a Binary compare on any of the lists it found with the fast methods (using the toolbar button at the bottom). It takes much longer to do this. FileDup may be doing this on the first pass which would explain why it is so slow.

      Ciao

    • alfonso

      Hi!

      I’m totally with Kevin here. I have exactly the same need. I’d like to set a master folder and check an other folder against it.
      Over time I copied my pics here and there and now I have a “pics” folder and a “pics dups” folder with twice the size, but I want to be sure I don’t loose any data!
      So, anybody heard about a software like that?
      thanks,

      alfonso

    • Minx

      I also have 6 externals and both the mac laptop and the windows vista desktop are near or at full capacity. Any recommendations on backing up the externals and checking for duplicate files on all of them. Do you prefer to partition the external and keep your backup of the external on the partitioned side or on an entirely different drive? Also get into trouble with compatibility issues with the externals being used on both the mac and windows even if they are formatted as Ms-Dos (FAT). Is it the file size? The files are mostly EPS, AI, PSD and Gifs that create the most trouble??? Is there any relation to the type of file? However, my biggest priority is cleaning up the unnecessary junk and finding and erasing duplicates. Its time to clean house! Any and all help is greatly appreciated. Have a wonderful day!!!

    • Terry

      Kevin and Alfonso – great idea and a big need for that one! I am currently sorting through software on an offshore drilling rig in the North Sea, and man do I need exactly what you have described in your request! Somebody through us a bone here.

      Terry

    • http://www.pilotman.com Kristian

      If you are looking for something working on folder level then try Clone Tools http://www.pilotman.com/

      With that piece of software you can compare folder content, merge folders etc. it will never delete any file that does not have an identical clone.

      The drawback is that it is not freeware :(
      Also it is a bit more complicated to use since it is more advanced then the programs in this list.

    • http://www.ashisoft.com RIZ

      So far the Duplicate Finder has worked great for me! It has been a terrific assistance for me in eliminating duplicate images and songs from my hard drives – saving me loads of time.

      http://www.ashisoft.com

    • neil

      This is the best free hassle free software i have ever used! Dont waste paying money for deleting files. This should be your one stop solution!
      Thanks for sharing the information, i could get back almost 5gb of disk space from my mp3 collection!

    • Thomas W.Adams

      Help For Windowsxp pro

    • dewey1973

      For those looking to compare two folders, try WinMerge:

      http://winmerge.org/

    • http://www.mindgems.com/products/VS-Duplicate-Image-Finder/VSDIF-About.htm Allan Cass
    • http://www.file-utilities.com Allan

      Try Directory Report
      http://www.file-utilities.com/downloads/wdir.zip

      It can find dupes based on the same
      name, size, CRC and/or comparing byte by byte

      Plus it can replace dupes by hard links

    • http://www.mindgems.com/products/Fast-Duplicate-File-Finder/Fast-Duplicate-File-Finder-About.htm Randy

      You miss the best of all and it is FREE. I guess because it is pretty new, but was featured at ZDnet recently:
      Find Duplicate Files Free – Fast Duplicate File Finder

    • Angelo Rigotto

      Buongiorno, รจ possibile definire una directory come master e cercare e cancellare i files nelle directory slave comparati solo con la master e non anche tra le directory slave.
      Grazie

    • http://indeepsoft.blogspot.com alex

      You can also take a look at Exact Duplicate Finder.
      If you want things to be simple, I mean real simple.
      If you also want to be in control, then give it a try.
      Press F1 to read help, after the app starts. Enjoy ;)

    • Ben

      thanks bro – was looking for somebody to “cut to the chase” and you did.

      appreciated

      “Duplicate File Finder” is absolutely roaring through duplicate photo archives here –

      much appreciated

    • Sgavhane33

      If you are looking for the solution to the duplicate files or image. You can use following utility (http://www.duplicatefilesdeleter.com/). its a good solution to the duplicate files.

    Copyright © 2005-2012 - Raymond.CC Blog