Update May 6, 2015: This script has not been tested with Mac OS X Yosemite 10.10.3 new Photos application.
Short Story:
I had several years of photos that I needed to identify and remove the duplicate. Instead of manually combing through 12,000 (read Long Story below) and before carpal tunnel set in, I needed a script to help me out. My situation may or may not be unique, so this script may not work 100% out-of-the-box for you, but it should get you started.
To use:
- Download and unzip the script
- Double-click the script to open in Script Editor
- Go into iPhoto and select a group of photos you want to compare
- Switch back to Script Editor and run the script
- Don’t Touch Anything! Just let the script finish, it could take a while if you are comparing a lot of photos
- After the script is done, go back into iphoto and search for “duplicate”
- You can highlight all the duplicates and delete them or move them some place safe
Photos are considered a duplicate if:
- both heights match
- both widths match
- the photo date in iPhoto match, this is typically the EXIF creation date
Long Story:
About a year ago I was editing down my iPhoto library of about 6000 images, just gitting rid of those out-of-focus shots and the ones of my wife’s feet (a curiously large number of these). After a long night of editing, the next morning I awoke to start again, but when I ran iPhoto there was nothing in the library.
It was all gone!
I couldn’t find anything anywhere. Could I restore from a backup? Ooh nooo. I had erased my backup drive the day before in preparation for moving the unwanted photos onto the backup drive and then making a new backup of my iPhoto Library. So I had no backup.
Not really funny. These were all the shots of my boys being born, first steps, first birthdays, first everything. I was up sh*t creek and it put a serious hurt in my stomach. At least I knew what to do: do nothing on the computer, boot from the Mac OS X install DVD and use Disk Utility to make a byte-for-byte copy of my internal hard disk. I could use this disk image to recover the images, hopefully.
So I tried several image recovery utilities and finally settled on PhotoRescue for Mac. I mounted the disk image of my internal disk and set PhotoRescue to the task. About 9 hours later (not a typo), PhotoRescue gave me several folders of recovered JPEGs, TIFFs, GIFs and PNGs. I tossed all but the JPEGs. I felt a little better at this point.
But when I looked in the JPEG folder there was over 12,000 images! Huh? Well, PhotoRescue does not discriminate, it recovers ALL images, including thumnails, web graphics, pron (you’ve been warned). Frankly, it was unbelievable and overwhelming.
So I set about dividing the images into folder that I knew were junk images and ones that I may want to keep. First, I eliminated everything below about 120K. I knew that my oldest digital camera was around 3M pixels and it saved a file that was typically > 200K so those images below 120K were most likely thumbnails and web images. That cut my stack almost in half.
Next I looked for images > 3M. These were corrupted image files that while they looked ok in Preview, I knew there was no way a 1200×1600 images was 40M. Just a consequence of PhotoRescue’s recovery routine. I can live with that, believe me. So I tossed everything > 3M because my current 6M pixel camera images are under 2M in size.
This left me with about 6,000 images that I imported into a new iPhoto Library. From the looks of it, all my images were there! What a relief, but the bad news was nothing was rotated properly, and there were many, many duplicates. Thousands of duplicates to be exact. After I rotated all the images so that I could view them properly, I set about removing the duplicates.
The good news about removing the duplicates was that they were fairly easy to spot. When I imported all the recovered images into iPhoto it apparently used the EXIF data data to date stamp each photo instead of using the photo file’s creation date, which was set by PhotoRescue to the day I performed the recovery. So all my photo’s were dated properly, I just had to look at each photo that matches (they were sorted by date) the one next to it and delete one of them.
A closer look at the duplicate photos revealed that while they had the same height, width and date/time, they varied in size. I was not able to determine why the file sizes varied as the images themselves looked identical, but my best guess is that the size difference came about from iPhoto’s insistence that when you rotate an image iPhoto considers this an “edit” and makes a copy of the original and add’s some iPhoto specific data (no verification on this though). So hey, if you are going to keep one, why not keep the smaller of the image files? So that’s what I was doing.
After hours and days of removing duplicates, I decided there has to be a better way. A bit of searching for “applescript iphoto remove duplicates” let me to Brattoo Propaganda Software’s Duplicate Annihilator. I tried the demo and it works very well. But there was one thing I wanted to do that Duplicate Annihilator could not, and that is mark the larger of the duplicate files. Duplicate Annihilator marks duplicate files by date/time which I am sure is what most people want to do. So definitely check it out.
So Duplicate Annihilator minor missing feature led me to write my own AppleScript to do pretty much the same. The script is pretty simple and requires no additional libraries or command line voodoo. But I will say that coding in Ruby for the past year-and-a-half really reminds my why I don’t like AppleScript. AS gets the job done, but it’s so much more work, frankly it’s confusing, and if you don’t do it often it’s a lot of work getting your head around AS’s nomenclature.
For you fellow rubists, there is rb-appscript which would have made my pain a little easier, but it relies on ruby and having the rb-appscript gem installed and that would be too much for most casual Mac users. So AppleScript won this round, but only because I knew I wanted to share the script for others.
Good luck to all you photo recoverers. I’ve been down your road before.
My apologies first as this may sound silly, I’m new to Mac.
I dont understand where to place the copy of the script. Do I open ‘iphoto’ and paste it in there somewhere?
I’m really needing a step by step guide.
You open the script in script editor, open iphoto and select the photos you want to compare (ie, you should see yellow border around pictures you want to compare). You can select all by going to ‘Edit’ and choose ‘Select all” from dropdown menu. Then, go back to script editor and press ‘run’.
I’m with those for whom the script *seems* to work (says it has worked) but then the iPhoto search doesn’t turn up any photos marked “duplicate.”
OS X 10.6.8 and iPhoto 08 7.1.5 (378).
ME TOO
Was there ever any feedback posted to Chris’ comment? I also cannot get iPhoto to recognize my searching for duplicate.
DOESN’T WORK – SAYS IT’S ‘RUNNING’ BUT THEN DOESN’T TURN UP ANY DUPES – WASTE OF TIME
Perfect! Thank you so much! You are wonderfull
Thank you. Works perfectly!
Hello,
I ran the script and iPhoto locked up and the screen “blanked” (i.e. everything vanished apart from the outline of the iPhoto page!) Had to Force Quit iPhoto. No apparent damage, but when I search for “duplicates”, I get the blank iPhoto page.
my start up disc is full due to the fact my iphoto has duplicated everything more than twice i ran duplicate annilhator but before i could delete them all my pics had disappeared and nothing was left in iphoto very annoying they are all in folders around the mac but i cant put them back as its full what can i do its very very annoying as it stops me from doing alot of other things clearly i am totally computer illiterate and am desperately needing some help ???
I have the same problem…Something keeps duplicating my iphotos…I have over twenty thousand pictures now…Probably three thousand pics are original
Can you tell me if your script will compare file sizes or just the physical size of the image? I am having an issue with Dup Annihilator marking some jpgs as duplicates and raw files as originals, even though the file sizes are different. Ideally I would like to be able to tell it to compare extensions not just just date/exif data, name and dimensions.
I am having a problem dragging my duplicates to the trash. I emptied my trash and that did not help. I cannot indicate trash for one or a group of pictures that are highlighted in the duplicates.
Hi Rosalie,
You can only trash pictures from the photos folder,not from albums…
hope this help..
good luck
Philippe
After the script is done, go back into iphoto and search for “duplicate”
You can highlight all the duplicates and delete them or move them some place safe.
This what I have done, but cannot trash them.
Hi!
Just to thank you very much for the script!
in about 2500 duplicate photos i could only point one not being dupe.
The only plugins or other software are not cheap, so i decided to give a try.
anyway thankfully apple decided to unify the libraries from iphoto and aperture!
so i imported all my photos to aperture, so that it moved all photos to folders by year/month/day.
then this “unification” allowed me to run the script on iphoto, flag the dupes, turn to aperture and delete all versions from the dupes (flagged) to trash, then empty it also moving the referenced files to the trash.(i did not find a way to do it in iPhoto)
now, i am only left to delete dupe photos with (1) or (n) on the name.
these photos because of different dimensions (thumbnails?) were obviously not detected by your script.
So thanks again Karl!
Thank you, worked perfectly 🙂
The script has been running for about an hour with no results. It just says “Running…” Is this normal? Thanks,
Erin
I am using your iPhoto script to identify duplicates. I keep getting message about apple event timing out. Working if I select a few photos to do but have 7,000 in lib
What am I doing wrong to get this message. I can leave it on all night to run if needed.
Hi there, thank you for the script. I need a little help…I’m getting an error because a message on iphoto is popping up saying it cannot find the original, and this in turn stops the script. “error “No result was returned from some part of this expression.” number -2763″ Can you help me?
perfect.. thanx a lot
I’ve been surfing on-line more than 3 hours today, but I never found any fascinating article like yours. It’s beautiful worth sufficient for me. In my opinion, if all web owners and bloggers made good content as you probably did, the internet can be much more helpful than ever before.
4 years later and the script still rocks – works like a charm and saves you $$$ – does what duplicate finder apps from the app store do for free.
not the fastes but works 100%
Thanks – Daddy rocks.
Thank you so much. Sometimes the simplest solutions are the best.
Well done and cheers!
-Jeff-
how long should this actually take. Im running it on 12000 images and its been going for like 3 hours. LOL =)
ok so once I have all the duplicates. Do I just delete them all? and it still leaves a copy in my photo’s? OR is this the original plus the duplicate in the folder?
One caveat. If you import both RAW and JPEG your script will consider dupes between them.
Here’s another caveat. When you use image path you’re actually getting the path of the preview file and not the original. You need to use original path to get the actual master file.
Hello Karl.
I have been using my iMac for about 5 years now, recently gave my iMac to my girlfriend and bought a PC. Trying to set up the PC with photos along with various backups i have made over the last few years has resulted in me having up to 6 duplicates in some cases! So i resolved myself to deleting every single photo individually! I had had enough of that after about 3 hours and so started to search online for a solution. Found this script and it has worked!!! Not only does this script mark the duplicate photos with the keyword “duplicate” but when i search in iPhoto for the keyword “duplicate” it only shows me the duplicated photos and leaves the original photo out from the search so i can easily delete thousands of duplicate photos!!
Thank you so much for being so kind as to share this with anyone who needs it. You are a kind genius!!
Thank you again
I have this error message? “iPhoto got an error: AppleEvent timed out.” number -1712 BUt it said that it had finished, I put in “duplicate” in the search box but nothing happened?
I got the same message but when i clicked search in the bottom of iPhoto and typed in duplicate, I had almost 1500 items show up. Some were duplicates but the system deleted both copies for some reason. I didn’t need the pictures anyhow but still.
You deserve some kind of Apple’s wage. 🙂 Thank you a lot! With this script you gave me more than 1 hour to play with my kids! Wonderful!
After using Iphoto AppleScript to remove duplicates I lost everything in images please help!
hi! looks interesting. trying it as i type. wouldu happen to have one for thumbnails?
this is self-less love for others… it’s Jesus Kindness… agape style even!
and a very awesome modern example of it.
thank you.
I’m not a programmer, but know enough to do serious damage.
Tried this script out on a few photos that I knew had duplicates. The script put “Duplicate” in the DESCRIPTION field under Information. I would prefer that “Duplicate” was added to the Keywords, because I didn’t want to over-write any existing Information.
So Searched around and found some references to iPhoto Keywords and tried modifying your script with:
FROM set comment of thisPhoto to “duplicate”
TO assign keyword of thisPhoto to “Duplicate”
Before I ran the script, I defined “Duplicate” as a Keyword in iPhoto.
Got an error for “to” (something about expected ‘given’, ‘with’, ‘without’ …etc). Don’t know enough to get any further.
If there is a quick response to this problem, please let me know. Otherwise, just drop this thread.
Thanks, Nik
I just tried this and get an Applescript Error of:
iPhoto got an error: Can’t get photo id 4.294996561E+9.
Am I doing something wrong? Using iPhoto 9.4.3, and OSC 10.8.4
Please help. Thanks.
Jeff
but what if you have more than one photo on a day as is always the case! does it use the time stamp to distinguish?
I’m runing this on 7000ish images…how long will this take?
This is an awesome script, it will help a lot with older photos. However, recently I tend to take some burst mode photos on my iPhone and they have the same time stamp and are the same size, but are different…subtle, but different. I found it kind of funny, guess you can’t compare water drop locations! LOL! I am glad you retrieved your lost photos…I keep an offsite backup. 🙂
Thanks so much for this script… I’ve been doing an epic cleanup of photos and ran the script with 22,000 photos… Yes… So it ran for 5 hours then I stopped it, went to sleep, saw the progress it had made then checked to see how accurate it was… So far so good, then ran it again in short bursts for smaller numbers of photos then a final 7 hours to pick them all up…
The accuracy for photos taken on iPhone like the comment above was something to note as photos taken very close together were picked up as duplicates… Also I had some photos there were really old that it picked up too, where the date had been changed manually as they were from film not a digital camera… All in all, I’d say 99.5% accurate and I had around 4,000 duplicates…
The reason for that many was iPhoto deleted a random set of photos in the middle of my collection so spent a while getting them back using Picasa, SD cards and got them all back and they’re now back in iPhoto…
This many photos I know now and have researched is too many for iPhoto so next step is to delete the images I don’t want and archive a selection of photos on a separate drive…
Am writing this as some people were asking about time it takes to run the script…
Hi Karl, I have run the program twice and gotten this message:
“error “iPhoto got an error: File file Macintosh HD:Users:JesseWeeks:Pictures:iPhoto Library:Originals:2007:Mission Music_2:Caribana 2005 002.jpg wasn’t found.” number -43 from file “Macintosh HD:Users:JesseWeeks:Pictures:iPhoto Library:Originals:2007:Mission Music_2:Caribana 2005 002.jpg””
I went in and was able to find the file so i’m confused. I tried deleting it and running it again but the same message came up.
Any help is appreciated.
Thanks!
Jesse
Thanks a lot, it’s great!
Hi Karl,
I’ve made some modifications/extensions to your script and i’d like to share it with the world. May i get your permission to post a modified version of your script? I’ve had a hard time finding your private contact info in order to request this.
Cheers!
~Bryan
I have 24,000 iPhoto images that the script is going through, and it’s been running for over 9 hours, so don’t expect quick results. Will report back if and when the script finishes.
You’ll need to only do a few at time for this script to work in a timely manner. I had it going about 20 hours on my large library and it was still running. In all that time, it managed only to do about 1300 duplicates out of thousands more. On a fast computer, too.
not working at all
This may not work with the new iPhoto in Mavericks. I can ONLY do a few at a time – almost not worth the effort. You then have to carefully keyword search the word “duplicate” to find them.
It doesn’t appear that the developer is currently monitoring this blog.
It was a good effort, though.
@Kurt: I do monitor the blog! Yes, the script is slow. It has never been fast. It’s a limitation of iPhoto’s implementation of AppleScript. I have tried it with iPhoto 9.5.1 (the most recent) and it works. As many users have reported, it can take HOURS to comb through a large library of photo. Breaking it into smaller batches helps the speed.
Thanks for getting back! I often get that time out error, and haven’t found the “sweet spot” of how many photos I can do at one time. What is your experience?
May I suggest changing the name of this great script to Iphoto Duplicate Finder. When you download “Iphoto Remove Duplicates” it’s intimidating (and not accurate).
Hi Karl: Thank you very much for sharing your script. It worked great and identified 950 duplicates for me. You have saved me hours of tedium. Thanks again.
Please help.
I downloaded script When I click on zip folder nothing happens.
.Can see the script emblem but there is nothing in the script box. What am I supposed to do to move the zip file into the script “folder”?
I just updated to Maverick and now have a lot of duplicates after iPhoto was updated too.
Can’t wait to use this.Thanks.
YOU ARE AWESOME!! THANK YOU THANK YOU! For years I’ve been manually doing this, but now I can get on with way better things…..
are the highlighted ones just the duplicates so iPhoto is saving one of all the highlighted? or do I have to uncheck one of the highlighted to save
Hello
thanks for a great script. Thus, it could be even better, if the script did compare the size of the pictures, or the type (jepg, mov, etc) . When uploading movies from my camera to iPhoto my camera always produces a movie and a picture of the first picture in the film. The script marks the movie or the picture as duplicate. Otherwise it just works perfect. Thanks!
May 5 2014 .. Just ran the script. When I entered duplicate it showed a lot of pics – 1 -2 -3 which I assume it has found to be to replications. I took the precaution of making sure I had a VERY recent back up and also turned time machine off so it didnt back up until I was sure that only dupes had gone. I am about to remove the -1-2-3 files. fingers crossed.
On latest Maverick version 10.9.2 and iPhoto v 8.1.2.
I did get a time out messsage >not sure if that was because Iwas AFk but I did get another small screen that seemed to indicate that all was ok . Fingers crossed. Will report back after dupes are removed
Good Morning, I am not sure if you are monitoring this page, but i had a question if you could explain the process…. why did you write the script to check height, width, and date only? Why not file name, date created and file size?
thanks for your time.
U Rawk! However, this doesn’t work for video files.
My IPad says ‘Safari cannot download this file’. Any ideas?
Tomo
Too easy, thanks very much!