For months I have been fretting about which application to use for my photo files. Because of the indecision I have also been worrying about archiving and data security. As our cameras gather more pixels and post-processing opportunities improve, data storage is a real problem. The new Sony A7r, for instance, produces RAW files of around 37MB while a processed TIFF from an original can approach no less than 100MB. It doesn’t take many days’ shooting to fill 15 or 20GB of disk space so deciding where to store current and archive libraries is crucial. As always, of course, security is the main consideration and more than one complete backup is essential. Ideally, one of them should be kept off side. First though, I had to decide which system, Aperture or Lightroom to concentrate on.
Aperture v Lightroom
For five years Aperture has been my application of choice for editing and filing my photographs. Lightroom, for me, is the new kid on the block. It comes free with most Leica cameras, so I started to experiment with it in 2012. As an Aperture man, I initially found Lightroom confusing and I had difficulty getting my head around the different method of file organisation. Aperture keeps all photographs within one file bundle which can be moved around from one location to another without causing problems. Lightroom, on the other hand, stores files in date order with reference to a main Catalog file which (in theory) remembers where everything is. Moving files around is more difficult but certainly not impossible once the basics are understood and reference links are restored.
In many ways I now warm to Lightroom’s file structure where photographs are sorted logically in year and date order. It makes for an easier overview of archives and, I suspect, is more future proof. Straightforward photo files will always be readable, irrespective of the fortunes of systems such as Lightroom or Aperture.
As far as post processing is concerned, I am still learning Lightroom and getting more confident with every shot I handle. I also make much use of Nik Software’s Silver Efex Pro for black and white conversions and for processing Leica Monochrom output. But this add-on works with both Aperture and Lightroom so it isn’t an important consideration.
My main gripe with Apple’s Aperture is the glacial rate of updates to accommodate new cameras. Adobe, on the other hand, are quicker but the software is also inherently more accommodating because of the ability to work with Abobe’s standardised DNG (Digital Negative) RAW files. For instance, after nearly a year Apple has still not issue support for Leica’s X Vario (which outputs standard DNG RAW files) so I have had to keep switching back and forth. Lightroom is also better at gathering basic metadata. It recognises Leica lenses, for instance (on the M240), whereas Aperture always returns “unknown lens”. Lightroom also makes a stab at estimating aperture which, of course, is not transmitted electronically from a manual lens. Lightroom guesses and it is usually not far out. All this is important when you use many different cameras and lenses as I do, particularly when you need to publish a photograph’s vital statistics.
All this tips the balance, quite irrespective of the competing methods of post processing. While I am still not fully comfortable with Lightroom, I have decided to use it as my main application for 2014. My Aperture master file is always available for past work and I see no need to export all that stuff into Lightroom.
Where to store data
There are two decisions on storage that need to be made, preferably as early as possible.
- Where to keep the current working files
- How to store archive files
Current files
Over the past two years I have changed my mind on current storage options at least three times. Until recently I was regularly using two computers, a 15in MacBook Pro Retina at home and an 11in MacBook Air for travelling. I hit upon the idea of using Dropbox as the location for my current Aperture file. Aperture, unlike Lightroom, doesn’t really mind where the files are stored or how they are moved around, so two computers can access the same data file over Dropbox without problems.
However, there are several real problems with this approach as I found out. First is sheer storage space. Dropbox storage is expensive. I have 210GB available (200 of which I pay for, 10 being a gift resulting from my recruiting efforts on behalf of Dropbox). Early this year I realised I had used 175GB of this space, at least 125GB of which consisted of the current Aperture feeder file. Worse, I realised that uploading a batch of photographs would keep Dropbox busy for hours, if not days. It always seemed to be churning away with implications for computer performance and broadband bandwidth. The final straw came on one occasion when I was away from home and relying on a slothful 2Mbps internet connection. There was just no way my 8GB photo update was going to be finished before I hit the road.
So, although initially a good idea, the use of Dropbox as a main location for photos (at least on the scale at which I manufacture them) is not really practical. And, while Dropbox is ultimately reliable with Aperture, I had constant problems getting it to work properly with Lightroom. Back to the drawing board.
I have now decided to return to square one and move my current Lightroom files to the home directory of the MacBook Pro. I am no longer travelling with the MacBook Air, so synchronisation of a second machine not longer vital. Instead, I will take the MacBook Pro for longer trips and rely on the iPad mini for shorter breaks. This simplifies matters enormously.
The retina MacBook Pro has a fast 500GB solid-state disk which is enough to accommodate up to 300GB of photographs in addition to the system files and minimal storage on the computer (I keep almost all my non-photo data files on Dropbox for ease of access and for immediate security and backup). The file structure of Lightroom makes it simple to copy files, month by month, across to external mass storage.
Archive storage
Currently my old Aperture and earliest Lightroom photos are on an external USB 3.0 Seagate disk which is backed up periodically to the old Drobo FS RAID which I am about to pension off. It is slow, very slow (working over the local network) and cannot compare to a local Thunderbolt drive or, even, the current USB 3.0 disk.
With ever-ballooning file sizes, fast and secure mass storage is now absolutely essential. All options are open. One thing for sure, I am done with Drobo because both the units I bought some years ago have been unreliable and far too slow. Instead, I am looking at Thunderbolt drives from LaCie or G-Technology. Both suppliers offer large capacities up to 6 or 10TB with RAID mirroring and the option to daisy chain more additional drives The LaCie 6TB 2big Thunderbolt drive is currently heading the list because of the ease in which either of the two drives can be hot-swapped in the event of problems.¹ This drive, which offers up to 3TB of storage (allowing for the mirroring) costs £450 from the Apple Store and, unusually, even includes a Thunderbolt cable which is worth all of £35. Three Terabyte is enough for the time being and I know I can always add another drive when it is needed. There is a downside and that is fan noise. These drives are noisy and, since Thunderbolt cables are short, there is no chance of hiding them away in a cupboard.
The big advantage of using one of these fast drives, which can achieve data transfer speeds of up to 400 MB/s, is that they can function as an extension to the computer’s internal disk. There is virtually no degradation in speed and this means you can access archives just as quickly as local files. Two years ago I was using a 256GB external LaCie Thunderbolt drive as main boot and data location for the old iMac. The SSD achieved much higher transfer speeds via Thunderbolt than did the internal mechanical disk. Boot times were slashed. The mass-storage RAID drives mentioned use mechanical 7,200 rpm drives but, thanks to Thunderbolt technology, offer similar speeds to conventional in-computer drives.
What is very clear is that it is vital to think through storage and backup strategy to ensure absolute security as well as speed and convenience. It is a truism that you can never have enough backups.
Apocalypse now
Imagine the worst that can happen to your data. Backups are one thing but, if you have enough of them, it is very unlikely that they will all choose to fail at the same time as your computer’s internal disk. So several independent backups, particularly of your main computer’s drive, are essential. I rely on Apple’s Time Machine (stored on a 3TB Time Capsule) but use it mainly for the recovery of individual files deleted by mistake. I have not had the experience of restoring a complete drive so prefer more traditional methods. In addition, therefore, I keep two separate cloned backups (with HD recovery partitions) so I have two faithful copies of the MacBook Pro’s disk on hand. One of these drives, a fast 1TB Seagate Thunderbolt, travels with me as a primary backup. The other stays at home. All my important data, with the exception of photo archives, is also stored on Dropbox so, in effect, I have a reliable off-site service in the event of the apocalypse. Cloning and scheduling of backups is handled by the excellent, robust Carbon Copy Cloner.
Backups are only part of the story. Think of a fire, a flood (particularly relevant at the moment in the UK) or, even, a particularly efficient burglar who runs off with your computer and every last backup disk you have so carefully hidden in the sock drawer. Quo vadis then? Up the creek without a paddle, without a doubt.
While I have my back covered as far as the MacBook Pro is concerned, my large photo library is still vulnerable. For the time being I copy the archives to a separate disk which I keep off site in a secret location (not to be divulged here. My lips are sealed). Yet this is a pain; something that I have to remember to do every month.
The alternative is a backup service such as Backblaze. Because these systems offer simply storage for emergency access (and not a live synchronisation system such as Dropbox), the cost is lower, about $50 a year for unlimited storage. The big snag is that the initial upload of your data can take weeks (depending on the speed of your broadband and, probably more important, the data limits imposed by your supplier). Once the initial upload is done the regular incremental backups should be manageable. For the moment, however, I am giving this type of storage a miss. It isn’t a question of cost, it’s the inconvenience and the difficulty of uploading that puts me off.
Parkinson’s Law
Just as Parkinson postulated that work expands to fill the time available, data inexorably expands to fill the space available. Back when the Macintosh was introduced thirty years ago one measly little Megabyte was both expensive and commodious. You could run a modest business on 10MB. Now we think of one Gigabyte as we used to talk about a Megabyte. Terabytes are commonplace and, pretty soon, no desktop will be complete without the odd Petabyte at its service. So, with these unbelievably large amounts of data in use, it is ever more important to think about the consequences of loss.
For my part, I rely entirely on my digital data. I am a demon shredder and every bit of paper that comes through my letterbox is quickly despatched after being scanned and optically recognised. Without my backup systems and regular tending of the data garden I could definitely be sitting in my canoe without a paddle.
Now back to taking photographs, something much more interesting
________________________________________
¹ After completing this article I did more investigation of RAID back-up options. While I did not change my conclusion about the effectiveness of the 6TB LaCie 2big Thunderbolt drive, I did acquire severe misgivings about the noise level from this unit. While reading a number of independent reviews, I decided I would more than likely regret having such a noisy drive near to the computer. Until we get longer (optical) Thunderbolt cables at a reasonable price, this is always going to be a factor. Instead, I settled on two 4TB Seagate Backup Plus USB 3.0 drives
and one Seagate Thunderbolt Adapter
Great article – I will have to evaluate all my options again as well