Finding and eliminating duplicates

One of the more difficult things about managing a large collection of video clips is understanding how much duplication you have. Duplication is not the same as being backed up. For true backup, you want the file to exist, at a minimum, on a separate device and preferably off-site. In general, having duplicates laying around just wastes disk space and your time. For example, if you are playing clips back to try to find a particular scene, it can be annoying to realize that you have already seen a particular clip before due to it being a duplicate. 

Videux can make finding and eliminating duplicates fast and easy.  Just click the "Show Duplicates" button at the bottom of the app, just to the right of the sorting controls. If your clips have already been "deep scanned" (more on that below), you'll instantly see all of the duplicate clips you have, displayed side by side. Note that they may not have the same thumbnail or even the same file name. Videux's duplicate detection doesn't care about that - it is doing a bit by bit comparison, so the results are when you have an exact duplicate. If the files have not been deep scanned yet, the scan will start; it might take anywhere from a few minutes to many hours depending on your library size and drive speed.  Videux is a multi-threaded app, and so you can keep working while the deep scan is in progress. My advice, though, is to start the deep scan in the evening and let it run through the night. By the way, you can stop and restrt deep scan at any time; it won't negatively affect anything. Deep scan is enabled or disabled in Videux->Preferences.

You can mark file for deletion within the duplicates view. You can even use all of the filters on the filter pane to reduce what you are seeing. If you filter out, say, one of the two videos in a duplicate pair, you'll still see the one that passes the filter. This makes it very easy to see where your duplicates are coming from. In addition, if you Ctrl+click on a duplicate, you'll get a menu option called "Select Filtered Duplicates in this Directory". A verbose menu item, to be sure, but useful... it selects all of the other duplicates show that reside in the same directory as the file you Ctrl+clicked on. By just pressing X, you now have selected those duplicates for deletion. They are not deleted immediately though - you still have a chance to change your mind. 

Duplicate detection is also useful for just the opposite intention... to verify that you have a hot backup of a set of videos. We'll write a separate post about to deal with that case. 

Deep scan - what does that mean? Deep scanning is our term for reading the entire video file (as opposed to "basic scan" that reads info about the file, plus generates a thumbnail, but does not read all of the video data). By reading the entire file and storing a checksum, we can quickly compare videos. You can see the checksum we generate over on the right side of the display under "Video Details". Duplicate detection is not the only thing we can do with this information though. In future builds, we'll use this to help insure data integrity over time. Deep scan can can time - after all, it has to read every gigabyte of each video file, but being able to verify an exact match is worth it.