Archive

Archive for August, 2016

Tip for downloading Picturelife RAW files from Smugmug

August 29, 2016 4 comments

OK, I’ve learnt this the hard way.

If you’re downloading Picturelife RAW files from Smugmug, while Down Em All seems the most efficient way to do this, you may find that your download times out.

Downloading multiple files simultaneously with down ’em all

 

This is because the URL for the file is created dynamically with a session key in the URL. Once the session times out – around two hours – then the URL is no longer valid. That means that of the 1200 RAW and AVI files I’ve queued to download, I may download 50-60 before the session times out and all the remaining files fail to download, as their URL is no longer valid.

What this means, is that I have to go back to the download page – logging back in to Smugmug first., otherwise that page is empty – re-open down ’em all, and then manually locate where it left off in the long list of files. That takes a while, but once done, I can Shift-select everything from the last file on, and continue.

So – what can you do better? Well – sort the list alphabetically by the description before you start the entire process. This means that when you have to go back and start again, you can at least find where you left off easily, by working by alphabetical/alphanumeric order.

 

Wrong – sorted by order in the download page

 

Right! – sorted by alphabetical file name

Advertisements
Categories: Uncategorized

Deduplication tools for Picturelife Photos

August 28, 2016 Leave a comment

A useful tool I’ve found for comparing Picturelife and SmugMug exports, is a duplicate file finder called FileMany.

I’ve usually used AusLogics Duplicate File Finder for much of this so far, but it only does bytewise comparisons, which fail when comparing Picturelife with Smugmug, because the EXIF data is different.

FileMany seems to be a hidden gem in that it can do comparisons by:

  • Filename only (including case sensitive/insensitive)
  • Hash
  • Bytewise
  • Image comparison algorithm

 

FileMany: Marking all duplicates in one SmugMug folder for quick deletion

 

It can also allow easy selection/deletion of all files in a particular folder. It’s not quite as elegant as AusLogics, but since it works with filename comparison whilst ignoring the hash, it means I can compare like-for-like directories one month at a time across PL and SM, and delete all the duplicates from one, in a couple of mouse clicks. Since I favour the Picturelife Export of the two, as it includes an XMP metadata file for each picture/movie, it’s easy to delete any matching SmugMug copies.

I’ve also been using Visipics for image deduplication when names and hashes are different, but it’s excrutiatingly slow to use. I’m hoping that this tool will do a ‘good enough’ job for exact image matches to be much faster than that, too.

Categories: Uncategorized

Downloading non-JPG files from SmugMug

August 28, 2016 Leave a comment

OK, I’ve found some remaining files in the SmugMug-supported method for RAW and other downloads

Going to http://www.smugmug.com/picturelife/download
after you log in, will give you a list of files to download. This does include movies that were successfully converted to MP4 and made available from Smugmug, so while again you will at least have these available, they’ll again be duplicates, perhaps without metadata, needing de-duplicating and filtering.

As they suggest, you can use Firefox and Down’Em All to download the entire page en-masse. To do this, start Firefox, install the extension, and right-click on the screen to select “Down Em all’. Note that you’ll need to set the Renaming Mask to *text* if you want to keep the original filenames rather than a pseudo-random hash.

Then you click Start, and let the data flow…

 

EDIT: While the ‘raw’ videos do stay in their original size and bitrate, they haven’t retained the ‘date created’ metadata, so you will need to identify and timestamp or file them manually.

For my RAW Nikon .NEF files, the EXIF metadata does contain the original “Date Taken”, although other datestamps for the file have been reset.

Categories: Uncategorized

Picturelife to SmugMug conversion – first problems

August 28, 2016 1 comment

So, here’s my current challenge

This is the total size of these three directories

Inside these directories should be three copies of the same photos:

  • picturelife_s3 is a download of my own S3 bucket, containing the raw data files used by Picturelife directly – such as including video thumbnails and resized copies
  • picturelife_sm is a download of the photos migrated to SmugMug, thanks to their recent on-boarding of the Picturelife business
  • picturelife_zip are the downloads I made using Picturelife’s exporter when it was working

Now, each should be the same – I should be able to just pick one, and delete the other two, right?

No. Definitely no.

  • picturelife_s3 by definition contains every photo Picturelife stored for me… but with hash (random) values for filenames, and often with none of the metadata like date taken, or caption, because all that data was not held in the bucket, but in Picturelife’s own databases
  • picturelife_sm is a transfer and export of PL’s photos via Smugmug… and while the photos are nicely grouped by month and mostly have original names and details, there are an awful lot fewer of them than in the other files, making me wonder if some were lost by Picturelife during their ongoing problems, or (perhaps less likely) lost during the Picturelife-to-Smugmug migration
  • picturelife_zip would be the ideal – an export from Picturelife in a supported manner, month by month, with full metadata such as date taken and GPS coordinates, plus name tagging, in an XMP file for each photo and video there. Unfortunately, I missed around a year’s worth of these

So – I need to figure out a way to sort all these out so that I end up not losing too many precious memories, and am able to find photos that I’m looking for!

 

And that’s what I’m currently working on. Unfortunately, it looks very complicated. My findings so far:

 

I definitely wouldn’t have deliberately deleted this Hot Air Balloon video

  • I typically find only half the photos/videos in Smugmug compared to Picturelife. I’m not sure if that’s because I deleted some when sorting them out between the PL export (periodically) and SM export (this week), or because of any database corruption.
  • I have definitely found some videos missing that I would not have deleted, such as a hot-air balloon flight we took (see still from the video above). Again, I suspect this is due to the evident corruption I’ve seen where a video is shown in the PL/SM database, even in the iphone app but clicking it fails to play it
  • SM appears not to preserve video timestamps – the date it was taken. You would end up with just a mass of videos organised only by the month the export folder is for. PL export did keep the datestamp, in the XMP file.
  • SM re-encodes video from MOV to MP4, at the same screensize, but a lower bitrate (which could be due to better compression)

 

 

The below table is a cut/paste of one I’m working on, where I’ve compared like-for-like sets of photos or videos between Smugmug and Picturelife. As you can see, I’m comparing various parameters – metadata, file counts, etc. As I learn more, I’ll share it here.

 

Month/Year

Description

SM Value

PL Value

Observation

Jan 2015

Total file size

527MB

1820MB

  

  

Total files

174 files

498 files

  

  

Total movie file size

17 files / 215MB

30 files / 1420MB

  

  

  


  

  

  

Video format


  

  

  

Photo list comparison


/

  

Sept 2014

Directory Contents

180 files / 400MB

846 files / 2660MB

  

  

MP4 files

17 files / 182MB

56 files / 2280MB

  

  

MP4 file : family present unwrapping sample


  

  

  

VIDEO0005.mp4 comparison

11MB size

30MB size

  

  

VIDEO0005.mp4 Encoding comparison


 
 

  


  

Smugmug version appears to have been recoded

  

VIDEO00005.mp4 Datestamp comparison


  


  

PL has original date in “Modified field” in both XMP file and MP4 file propertes. SM only has current

  

May 2014

IMAG0528 Properties comparison


 
 


  


 
 


  

  

  

IMG_2331.MOV


 
 


  


 
 


 
 


 
 

Photo details from the same event:


  

Again, the SmugMug video has been re-coded:

  • to MP4 from MOV
  • From 22Mbps to 7mbps
  • From 29fps to 30fps
  • From Mono to Stereo sound
  • From 55MB to 20MB

 
 

The datestamping is interesting:

  • Smugmug has no original datestamp at all:
    • the modified date is the date it was imported from Picturelife
    • The created date is the date it was downloaded from Smugmug
  • In Picturelife
    • The file’s “date modified” is 5th June likely the date the download was created from PL
    • The XMP file’s “date modified” is 10th May and appears to be the original date the photo was taken

 

 

 

 

Categories: Uncategorized

It looks like Picturelife to Smugmug is working…

August 25, 2016 Leave a comment

So, I clicked the link from Smugmug, created an account, and it imported my Picturelife photos to it. At first glance, at least.

2016-08-25 21_34_25-Clipboard

Could it be… all my photos?!?

Now, I’ve not done any quantitative assessment of this yet, but first impressions are encouraging:

  • The organisational structure is by year/month, just as for the Picturelife exports.
    If you had gone to the Picturelife export page, you would have seen the chance to create zips for each month. This looks like the same organisation. It should make it straightforward to understand which photos you have, and which you don’t – say, if you used Picturelife exclusively between cetain dates.
  • The metadata appears to be preserved. Photos have metadata dates that match when I remember taking them. I saw a few have captions that I added in Picturelife. That’s good. [Edit: the help page notes titles/captions have copied, but comments have not].
  • The process works on self-hosted S3 buckets too. I have a self-hosted bucket, not Picturelife-provided. Smugmug imported all (?) the photos OK. And I can log into my bucket and see my photos still there.
  • You can download a zip of your photos by album, one album at a time. It’s a bit slower than doing the whole lot at once, but perfectly fair – it’s exactly how Picturelife permitted it, and each zip could be hundreds of MB or GBs as it is. It even does the zipping in the background and mails you a link, like Picturelife did.
  • The errors seem to have been copied across too. I found some problems, such as a movie thumbnail that showed one image, but the movie that played was completely different. Or Picturelife failed to find the movie at all. I had found that in Picturelife, and I’m finding the same in Smugmug. I’m not sure if they’re exactly the same errors, but I would presume they are. I’m not sure if these errors crept in when originally being uploaded, or during some ‘glitch’ when Picturelife was on the ropes. Either way, it looks like those movies are gone. I should be able to check if they’re really gone in the original S3 bucket.

Generally, it seems at first glance a very clean migration. The organisation is much as was in Picturelife. The download process is as in Picturelife. I presume the downloads will include XMP metadata. This is probably the best possible outcome, if all your photos are there.

 

And how about the 14-day Smugmug free trial? Am I going to stay with them?

Well… we’ll see. The service seems very similar to Picturelife, albeit targeted at photographers selling their photos, rather than family sharing. Having said that, they seem to cover that too, as they other users, comments, link and album sharing, sharing by tags, etc.

However, it does lack some of the tools of Picturelife. They have all the usual transforms, crops, rotates, and a more detailed editor – PicMonkey, rather than Aviary- but it’s slightly harder to access, and slower. The iOS app appears less functional – it’s beautiful, allowing viewing and link sharing, but no editing. Both are missing the calendar, the map view, face tagging, and (in iOS) the search function, which are big lossses.

I think most ex-Picturelife users will be happy with Smugmug, and continue to use them. I’ve moved my process to Mylio now, so I’ll be a little more cautious, but heck, I may just continue paying Smugmug while I make up my mind.

 

EDIT: Note the Smugmug, FAQ for the migration here, including some ‘hidden pages’ to access things like RAW files. The more I read, the more I am impressed.

Categories: Uncategorized

Photolife migration to Smugmug starts

August 24, 2016 Leave a comment

Today, within a couple of days of the Picturelife, I received my Smugmug invitation as promised.

Let’s see how this goes

Categories: Uncategorized

Picturelife migration to Smugmug

August 22, 2016 Leave a comment

So Picturelife is finally, definitely, conclusively shutting down. I just received this email. 

TL;DR: they’ve migrated all customer photos to SmugMug, and agreed a free customer export package with them so customers can use SmugMug as a front end to export their photos (or subscribe to SmugMug going forward).

Dear Picturelife Community,

After four years in business, we’re saddened to announce the end of Picturelife. We’ve spent the last few months trying to find a solution to keep the service alive but we’ve had to face a challenging economic environment and a decreasing quality of service.

Fortunately, we’ve had the opportunity to meet with the team at SmugMug, a highly regarded company in the photography industry. A few weeks ago, we reached an agreement with SmugMug to provide you with a way to recover your photo and video memories. For those of you who don’t know SmugMug, it is a profitable family-owned, 14-year-old business that helps photographers at all levels safely store and share their photos. Although their service is not free, they have graciously agreed to offer all members of the Picturelife community access to their photos and videos for absolutely no cost or obligation. You will have the opportunity to download your photos and videos for free, and then decide if continuing with SmugMug is right for you.

As of today, your memories are now safely stored on and accessible through Smugmug, while picturelife.com and Picturelife’s apps and support are all being discontinued effective immediately. Our goal has always been to provide a safe place for you to preserve your photos and videos and through this agreement we are making sure they’re still available to you.

Look for an introductory email from SmugMug in the next few days that will contain all the information you’ll need to easily access your photos and videos. In the meantime, please visit smugmug.com for more information on their services.

We’re honored that you’ve trusted us with your memories, and your enthusiasm towards the community was a source of constant motivation for the whole Picturelife team.

Thank you again for using Picturelife, and for all the love and feedback along the way. And more than that, thank you for letting us be part of your life for some time.

Jonathan Benassaya

At the same time, I’ve just had a comment from a fellow PictureLife ‘victim’, who said they recovered 50% of their photos using this script.

So which to use? Well, I’ll probably try both, then download them all to home server and check for which is the most complete.

  • I already have a PictureLife zip download (the best previous method) for some photos, but I’m missing around a year.
  • I also have a full export/dump of the files as stored on S3, since I hosted them on my own bucket, but while the images are there, the metadata ain’t. That’s a long slog of re-tagging still to be performed, with many issues such as incorrect original EXIF dates.

So – I might use the other two methods, and then at least I have a copy of all my photos in some form.

The trick will then be finding enough rainy afternoons to analyse and deduplicate them all.

Whichever you use, just remember to learn from your PictureLife experience:

  1. Use a service that permits full, regular exporting of your photos
  2. Remember to actually export them
  3. Remember to back up your export somewhere else