In this article Paul Dymond (dymond.com.au) will be walking us through his experience with harnessing the power of catalogues as a travel photographer, and also his skills with helping other organisations to get better organised themselves to make the most of their in-house image libraries. I’ve asked Paul to think about several different scenarios when covering the topic of catalogues:
- You’re a prolific but disorganised photographer with a lot of RAW files lying about on hard drives and haven’t processed them all or even indexed what you have collected.
- You’re well organised photographer who keeps on top of RAW file processing but need a catalogue to organise sets and collections for presenting or sharing.
- You’re a small company who hires photographers to build an image library but organising those assets in the office is a nightmare.
Get Organised
If there is any one thing that bonds and unites all photographers it is the phrase “I really must get around to sorting out my photos”. Oh I wish I had a dollar for every time I’ve heard this. Only very few people ever seem to get around to it. It gets put into the too hard, too time-consuming, too confusing basket.
Going from zero to having a well organised photographic archive can take quite a bit of time, but it will save you much more time over the long run and it isn’t anywhere near as difficult as people assume. Take it from someone who has a fully catalogued archive of nearly two decades of professional shooting as well as extensive experience building similar systems for some very large organisations, you can do this. Imagine the joy of being able to find any image you want in seconds rather than hours or days. Let’s get organised!
Not everything that lets you see your pictures can be classified as catalogue software. Browser software is what many people use to view their pictures. Something like Windows Explorer or Adobe Bridge. You click on a folder and it shows you the images that are in that folder, and only that folder. Any changes you make to images in this interface will affect the encoded metadata of the original image, and subsequent copies.
Cataloguing software, on the other hand, lets you import images from multiple folders, on multiple hard drives, and have them all appear on the one screen. When it ‘imports’ the images it doesn’t move them, it merely plots a path to where the image lives on your computer. Most of the changes you make to an image within cataloguing software are virtual, meaning that they only live within the software itself and don’t affect the original RAW file or image.
This is a very important differentiation as we will see.
Key Considerations
Keywords are really handy for the kind of classification fields which are permanent, generic and easily input to large number of images at once, such as names of people and places. Many software tools (including Capture One and Lightroom) allow you to add metadata at the point of importing into your computer or exporting JPEG from RAW. So if you’ve just been on holiday to Paris you can embed the word Paris into all the pictures you took as you transfer them over to your computer. But scrolling through 3000 images of Paris to find the ones you took of the Mona Lisa in the Louvre doesn’t sound like a lot of fun.
This is where detailed keywords of things such as location and names of people become very handy. If your software has a function known as hierarchical keywords and auto-predict you can even type the first word in a place name and it will fill in the rest for you. In this case you need to start with the tiniest detail first. For example if you were to label your photos of the Mona Lisa in this way: Europe, France, Paris, Louvre, Mona Lisa, auto-predict wouldn’t work too well. You would have to type in basically the whole string before you got to Mona Lisa. Whereas if you entered the information in the reverse order (Mona Lisa, Louvre, Paris, France, Europe) as soon as you type in ‘Mo’ auto-correct would most likely pick up the rest and automatically fill it in for you.
The biggest limiting factors with keywords is time and return on investment of that time. Stock Libraries add at a bare minimum 50 keywords to every image because they want their clients to find the right picture. So they will keyword things like concepts (happiness, family, traditional), colours, shapes. You name it, they’ll keyword for it. Or rather, the photographers who submit the images will keyword for it. It really is a time-consuming and boring process. You have to put in multiple spellings of words (for our North American friends who like taking the U out of words!) Add in plurals, latin names of plants and animals, synonyms - any keyword that you think will help clients find your picture. When keywords are the only way you can find a picture this is a necessary evil, but obviously far beyond what an individual photographer would ever need and this is where an alternative way of organising images is needed, one that is quick and easy, visual and virtual.
Get Sets
Software manufactures that promote "helping you find your pictures” is a pretty low bar to set. Lots of software packages can help you find your images, assuming you have keyworded every single one of your pictures that you have ever taken. Just type ‘elephant’ into the search box and voila, all your elephant photos will pop up on the screen. But what can you do with your pictures once you manage to find them? That’s where the rubber hits the road for a good catalogue.
Keywords are often heralded as the be all and end all of finding and organising our photos, and they can be very helpful but are far from the only, or often even the best way of organising your pictures. Instead of relying solely on keywords, temporary collections of images is often a far more useful tool. Take for example when you want to gather an editorial collection for a magazine that spans several countries - you don’t need keywords here, you just need to define a set of images for that particular purpose.
Capture One does this using Catalog Sets, while Lightroom implements Collections. Whatever the name, they are the same thing. Virtual groupings that only live within the catalogue software, into which you can quickly drag and drop from one to a million photographs in the blink of an eye. If your software has the capability you can have what are called nested sets, or sets within sets. One picture can live in as many sets as you like, just drag and drop, but because it is all virtual it means that nothing changes with the original image, which is nestled safely on your hard drive. You haven’t duplicated it, or moved it, it doesn’t physically exist in multiple folders, yet virtually it can exist in a hundred different groupings.
Next I want to lead you through an example of how I make use of virtual groupings in preference to metadata and keywords.
My Catalogue Workflow
I spend time every year in my wife’s home town of Sapporo, Japan. So over many years I have accumulated tens of thousands of images of this part of the world. Let’s say I take a photo of a group of young Japanese girls in kimonos at a summer festival known as Sapporo Matsuri. When I import the pictures into my desktop computer I will enter basic metadata, in this case location data of Sapporo, Hokkaido, Japan. That’s all.
Where the images live on my hard drive is basically irrelevant but for convenience sake I label my Raw file folders with sequential numbers which makes it easier to check that my backups are all up to date. So, boring as it may be, my folders are named Raw_001, Raw_002 etc. Within those folders might be subfolders with slightly more information about where the pictures were taken, something along the lines of Sapporo Holiday 2020. My files are named with my surname, the date taken, and a unique identifying number, all of which my software does for me in a Batch Rename. So a photo name might look something like Dymond_200323_1234.cr2. So no specific information about the image in either its file name, or the folder name where it lives. All that information lives either within the metadata of the picture, or virtually within my catalogue software.
I then tell my catalogue software where the new images are and it imports them all in. Just to reiterate, the “import" doesn’t move any of my files, just create a file path to the originals and builds a little thumbnail for preview purposes. By double clicking on the thumbnail I can jump to the original image at 100%. Even if I import multiple folders of images at the same time, all the images will appear on the same screen. That means if I photograph one location on multiple days, and even if those images live in different folders, my catalogue software will present me with a single pool of images for that location.
So now I have a few thousand newly imported images of Sapporo and it’s time to sort them using my virtual groupings. I have a Catalogue Set called Japan, and within that I have subsets for Hokkaido and Sapporo. Within each of those general location sets I have further subsets such as food, festivals, traditional dress. You name it, you can create it but just remember that too many groupings means you’ll spend more time clicking on boxes than actually looking at pictures. I find that I tend to limit the number of images in a subset to around 100 to 200 pictures. This enables me to quickly scroll through and find the one I’m looking for. Too much of a good thing can still be bad for you.
Let's go back to our image of the young Japanese girls in kimonos at the Sapporo Matsuri, as this one is remarkably easy to catalogue. It just gets dragged and dropped into the Sapporo Matsuri subset of the festivals subset of the Hokkaido grouping. In fact I select all of the pictures I took at the festival and just drag and drop them all in one go. No mess, no fuss.
Could I have keyworded the picture with terms such as girls, kimono, festival, Sapporo Matsuri? Sure. Could I be bothered? Most definitely not! Do I have to in order to find the picture again? Again, most definitely not. All I have to do is click on the Sapporo Matsuri virtual grouping, do a bit of scrolling and there it is. And for the vast majority of my pictures this is all I need to do…until it comes time to share those pictures.
ReCopy
Output is just as important, if not more important, than input. Part of my business is to help large organisations get a handle on their huge image libraries. I spend months sometimes sitting in front of their computers importing and cataloguing their tens of thousands of pictures but before I even begin there’s a really important job: Deleting duplicates.
The most common way to share a picture in anything other than its original size and format is to open it in your preferred photo manipulation software, re-size it, possibly re-name it then save it to another folder on your hard drive ready to be emailed or burnt to a hard drive and sent away.
So now you have two copies of your photo, stored in different folders on your computer with possibly a different name and nothing connecting the two except your (sometimes unreliable) memory. A couple of months later you need to send the same image to another person, you forget that you already did it once, and yet again you re-size it, stick it in another folder and send it away. Now you have three copies of your image! And so on and so forth until, just like the Coronavirus, exponential growth means you have multiple copies of multiple images clogging up multiple hard drives on your computer.
My record is twenty different 120MB files each with fifty exact duplicates living on a client’s network.
Remember the difference between our browser and our catalogue software? The browser only shows you what is in the folder you’re looking at. Your catalogue software can virtually show you images from different folders all on the one screen. Good catalogue software will render your need for duplicates obsolete.
Sharing not Duplicating
We return again to our image of the girls in kimonos at the festival which live in my Sapporo Matsuri virtual grouping. Let’s say I want to post some of those images to Facebook, or send them out in an email. The first thing I would do is create a virtual grouping and name it ‘Sapporo Matsuri Images for Email’ and then just drag and drop pictures into that grouping. Remember, they’re being added to this new grouping without being removed from any of their original groupings. They now appear in two places (virtually) but physically there is still only one picture on my hard drive.
Now I have a temporary grouping of images that I want to share. For many software packages all I have to do is select all the images and hit the email button. The software will duplicate the images according to the specifications I input, re-size them, attach them to an email and then open the email ready for me to send. No opening up of original images, re-sizing and saving in a different folder. No duplication. No clogging up of hard drives. Just a simple one-click to send them all at once. Again, just to beat this point over your head with a hammer, these can be images stored in multiple folders on different hard drives. The software will reach out to them all, wherever they are and attach them all to the one email.
Lightroom even has a function where you can link it to your social media. Select all the images and, for example, click on the Facebook button. It will go and grab them all, re-size them and open Facebook for you to post. It couldn’t be easier.
There are times, however, when you want to send a large number of images on a hard drive, or upload them to Dropbox. In times like this we might need to duplicate the images, but using our virtual groupings means we only have to duplicate them temporarily.
Take the case of entering a photography competition. We create a virtual grouping called Photo Comp Entries and we start curating a collection. Over a period of time we add and remove images until we’ve come to a collection which we’d like to enter. They have to be uploaded to a website at certain dimensions.
The exact route we use here may differ from software to software. Some software may let you click on the images and then convert them from within the software itself. You just input the dimensions, file size and format you wish the pictures to be then create or select a destination folder. Click, click, click and you have duplicated the images. Other software might require you to open them in photo manipulation software. This will often be a simple right-click or drag and drop operation. In this case you will go through the same old process of re-sizing and duplicating the pictures but the virtual nature of your grouping means that images from multiple folders can be opened in your manipulation software all in one hit.
So now you have a folder of duplicate images. Just what we don’t want! Never fear, we’re simply going to delete them from our computer once they have been delivered to the competition. Abracadabra, gone. Wait, what? But if we delete them what happens if for some reason we need to re-submit them? Then we have to start from scratch by trying to remember what images we chose and then start from the top. Or do we?
I keep these “temporary" groupings forever. I never delete them. I just leave them to live their lives happily and virtually within my software. That way if I ever need to create the same grouping I just have to select them all and hit my Duplicate button again. Hundreds of images can take less than 10 seconds. That means if I send a bunch of images to a client, which they receive, I then delete them. If the client calls me back a week later to say they’ve lost them it’s a really easy process to do it again.
There are many cases where it can be handy to see which images you used in a previous instance. I’m often asked to speak and present images at camera clubs and other events. I will create a collection of images for the talk at a camera club, and then if the same camera club invites me back a couple of years later to speak again, I can take a look at my virtual grouping from the last talk and see which images I presented to make sure I don’t show the same images twice.
I consciously make the distinction between permanent catalogue sets such as “Japan” or “Sapporo", and the utility catalogue sets such as those I create for a publication, or a camera club or any other set where the purpose of the grouping exists outside the content of the photos themselves.
The RAW Deal
Some companies don’t play well with others as we well know. This can present us with some hurdles within our Digital Asset Management framework both on the input and the output side of things. Firstly if you shoot only JPEG then you don’t have a worry on the input side. If, like the vast majority of professional photographers, you only shoot RAW then there could be issues.
One of those companies that doesn’t play well with others is Adobe. When you make changes to a RAW file in Adobe Camera RAW (ACR) the information is written to a sidecar file. This important little file contains all the information about any changes you may have made to your RAW file. As long as you’re opening that file in Adobe software you can see all those changes. The problem occurs when you open the file in a third party software, such as cataloguing software. Chances are you won’t be able to see any of the changes you have made and the thumbnail will only show the SOOC (Straight Out Of Camera) RAW file with no edits visible. Totally useless in other words.
The solution is to convert your proprietary RAW files to DNG files. It sounds like a complex move but in reality it’s effective and practical.
My workflow is to copy my images from my camera cards to my computer, make my changes in ACR and then save them all as DNGs. I then get rid of all my CR2 (Canon RAW) files. I import only the DNG files into my catalogue so that I can see any changes I’ve made, as well as any updates I make to my edits as I go along. Keeping within the same software family (Lightroom and Photoshop, and using only ACR for RAW processing and cataloguing) obviously solves this problem. If you choose to use a standalone catalogue software this could be an issue and worth investigating during the free trial period.
The net result for me is not having any JPEGs in my library. Everything is RAW in the DNG format, and when it comes time to send files to a publisher Media Pro does a fresh conversion from the DNG saved in the most suitable format for them. Once the publisher has the files I delete them from my file-sharing space. At my end at least, not a single JPEG remains.
Time Travel
The main objective in the creation of a catalogue is to do it once and live with it for as long as humanly possible. It’s a big job initially and you don’t want to have to repeat it ever again. This means that you need something that is pretty much infinitely scalable (as the number of images in your collection grows), and the information contained within it needs to be accessible even if you decide to migrate to another software solution.
In terms of keywords that’s usually not an issue. The keyword section of metadata is readily transferable, even if you have to side-care the information between catalogues.
"Catalogue Sets" however are less easily migrated. These are proprietary as well and very rarely translate from software to software. This presents us with an issue if we decide to move our neatly curated archive to a different programme, or if, goodness forbid, the company discontinues the software. (yes I’m looking at you Capture One MediaPro and Apple Aperture!) The last thing we want to do is lose ten or more years worth of thoughtfully constructed Catalogue Sets.
All is not necessarily lost, however, just time-consuming. Many software programmes have a rabid fan base that create droplets and scripts to enhance the capabilities of their products. One of those capabilities is the ability to add Catalogue Sets as Keywords to images in batch (I use this function a lot with clients). And many standalone catalogue programmes will recognise those virtual sets. The trick is to find which programme recognises the virtual groupings created by the software you use.
Of course this is all moot if you use the same programme forever, but in a world where many older RAW formats are no longer supported by camera manufacturers, assuming that your catalogue software will always be available could be a risky assumption. I want to make a special mention here for users of Adobe Lightroom, because the subscriber model is an issue for some photographers. In the event that you unsubscribe from your Creative Cloud subscription with Adobe you are allowed to continue using the Library function of Lightroom. For now at least.
From Catalogue to Media Library
The one downside of most catalogue programmes is that they are really designed to work on individual computers, not on servers, and they mostly make it finicky to work with multiple team members. It can be done but it needs a bit of planning and forethought to set up a system that works. Lightroom, for example, won't let you park your catalogue file on a server. You can only save it to an individual drive. Other programmes will let you open the catalogue on a server, but limit it to one user at a time.
One workaround I have found is combining Lightroom with Dropbox, and this works especially well for my clients who have a media library of processed images (JPEGs instead of RAW files) and need to catalogue them for distribution and usage. If you save a Lightroom Catalogue to your Dropbox folder it can then be opened by anybody who can access that same Dropbox account. You can share both entire catalogues and sub collections within a Lightroom Catalogue, by making use of the export feature. These can be exported with or without the images embedded in them. So you could have your original hires images on a local server accessible by different users, with the catalogue of those images living in a Dropbox folder.
There are many ways to cut it when all you need to do is catalogue finished JPEGs.
The number of images in each catalogue and the pixel size of the preview will determine the final size of the database file. If you don’t require high quality previews, just a few hundred pixels, then the catalogues will be manageable to shuffle about with conventional cloud storage like Dropbox, Google Drive or One Drive. However, If you decide you want the previews at a size to match your screen resolution, to accommodate detailed examination of each image in the catalogue, you end up creating a significantly larger database. Every slight adjustment to even a single file in the catalogue means the entire database will have to re-sync with your cloud storage.
I usually recommend to set up multi-member teams by providing them each of them with a copy of the same catalogue, prepared in advance with “Permanent" virtual groupings inside. From that point forward individual members can create their own Utility groupings which will only be visible to them in their instance of the catalogue, unless they share them with the other members of the team. This allows each team member to work on their own projects without the need to share everything with the other members of the team.
Find Me Online
If your budget is big enough there are a small number of enterprise level products that take the idea of Digital Asset Management and scale them up as only “the cloud” can do. Cloud based DAMs can be great for setting up brand assets for large companies, and distributing sets of images amongst their teams or partner agencies. These tools also focus on more than just images, embracing video and graphics files with equal importance. For this article I’m leaving the enterprise options to one side, and want to focus on an individual scale instead.
These are more Media Libraries than cataloguing tools, with an emphasis on image sets and key-wording being done by teams.
The downside of this approach is that they are often heavily based on the use of keywords and everybody needs to be on the same page when it comes to terms used. If one person uses the word 'family' and another uses the word 'families' then any search for pictures of a 'family' will only uncover a small portion of the images. Virtual groupings solve this problem, but again they need to be created by one member of the team to ensure consistency.
PhotoShelter has a product called Libra, a very intuitive and comprehensive cloud-based catalogue system.
Websites such as PhotoShelter are always looking for new ways to be more useful to their customers. These are powerful websites, full of features for presenting your work in collections and sets with support for RAW files. Delivery of hires originals and selling your pics is also part of the offering. For just USD$50 a month you can store as many images as you like, making it a powerful platform for selling work from very large collections. As well as housing my image collection, this is my preferred method of sharing Utility sets with clients. I will create a Utility set in MediaPro and then upload all those images to PhotoShelter in a folder with the same name as the Utility Set. Then it's a simply matter of sharing the details with the client to download the images at their leisure.
In the context of cataloguing your work, however, the idea of using such sites is very much a case of putting the cart before the horse. You really need to catalogue your archive before uploading them. Unless you intend to upload everything to a website you can’t take full advantage of creating catalogue sets. And if you do upload everything these sites may not be very effective at browsing and sorting at such scale. Logically speaking you could duplicate your entire catalogue structure, including nested sets, on a site like PhotoShelter but realistically it would be very time consuming and difficult to maintain as you make changes to your desktop catalogue.
And there’s one more issue which looms over pretty much any cloud based service: we have no idea how long any website will stick around. We’ve seen this already for standalone applications that migrate away from their original customer base, or evolve into something completely different. At least with a paid-up application sitting on your desktop you have the lifespan of your computer as a window for seeking alternatives. Websites can disappear overnight, either when a company fails or when you decide to stop paying the subscription.
Soft Solutions
Keeping track of your images is just as important as backing them up. What’s the point in taking all these beautiful photographs if you can’t find them or share them? The question isn’t whether you should purchase catalogue software, rather which tool will suit you best.
Use this article to guide you on what features matter most. You'll want something to let you keyword pictures (they pretty much all do that!), preferably with hierarchical capabilities as well as auto-fill to save time. You also will want virtual grouping capabilities to let you quickly and efficiently group large numbers of images for use in both permanent and temporary project groupings. Not having this capability is a deal breaker for me. Finally, you need to look at the output capabilities and whether or not you can output your tuned up RAW file direct to a suitable JPEG format.
* Can you send images with one-click?
* Can you duplicate images from within the software or do you need to open them first in your photo manipulation software?
* Do you have direct access to your favourite RAW processing tools if you want to re-evaluate the processing of that file?
This might then determine which software is best for you. If you already use Photoshop then Lightroom might be the software for you, and indeed is very popular with many professionals. If you are a Capture One user then you will already have found the catalogue capabilities within their RAW workflow tool. It’s extremely powerful for loading metadata into your RAW images, but now you can appreciates there’s much more power in a catalogue than just the keywords.
Other standalone catalogue software includes Photo Supreme, iMatch, PhotoMechanic 6, Neofinder, Fotostation and DigiKam. My choice of software is called MediaPro, which was recently discontinued by Phase One. When they originally bought the MediaPro software it was a great companion to their RAW workflow solution called Capture One, but over time has been integrated into that workflow to keep pace with Lightroom’s library feature.
I live in hope that a standalone version will be resurrected but, until that happens, I’ll be keeping an eye out for something that can match it. So far that has not happened.
Getting started with cataloguing can be a daunting process but thoroughly worth it. Besides, if we’re all stuck in our houses for the next six months what better to spend that time than reliving the joy of those photographs nested deep inside those hard drives?
