Updating my photo import pipeline
Why?
Recently I took up photography again. Or rather, I made more of a conscious effort to take pictures again, as opposed to snapping them. I have my existing image import pipeline, which copies the files from the devices to my central server into timestamped directories.
Recently I actually exhausted the storage of my mobile phone and that made me
realize that I want to run my photo import via rsync from abroad on my phone
instead of local-only copying via mounted drives.
Reading files
Before the change, the program took all files in the DCIM directory and
copied them via an SMB mount to the central server:
- Find all files in
DCIM/Cameradirectories - Read the image creation times from image EXIF data
- Upload all files into directories grouped by their timestamps
- Move all files into the
DCIM/Camera/archiveas a safety measure. I manually delete thearchive/directory from time to time
But running the import on a gphoto2 mounted directory does not allow to move transferred files to the DCIM/archive directory.
This would be ameliorated by using rsync. but a more discerning approach is still needed since I might have renamed the target directories
on the central server in the meantime.
The new approach addresses these points
- Find all files in
DCIM/Cameradirectories - Find (rough) creation date from image filename (fast), or from image EXIF data (slow, needed for sequentially numbered image file names)
- Read the remote directories starting from the earliest image timestamp
- Associate each image with a target directory
- if it already exists on the remote side, use that directory
- use the image timestamp for the image otherwise
- Upload all files into their target directories using
rsync
Using rsync and always transferring / synchronizing all images makes the transfer resilient against network failures and partial transfers.
Looking at the already existing filenames on the target end also opens the door to reading calendar entries and creating the directories with their calendar entry outright, as I've decoupled the directory name from the image creation timestamp.
Having an ssh remote target
Moving to rsync also enables backing up to a remote machine over ssh. To move the files into the appropriate locations, I also need to read remote directories.
I did not find an established module on CPAN for this. Filesys::Virtual::SSH is OKish, but I don't like the API
of Filesys::Virtual overall. I only needed a single routine to read a remote directory, so I wrote a very simple wrapper:
sub r_readdir($dir, $type="d") {
if( $dir =~ m!^ssh:(?<host>(\w+\@)?[^:]+):(?<path>.*)! ) {
my $p = $-{path}->[0];
my $cmd = "ssh '$-{host}->[0]' 'find \"$p\" -type $type'";
currently("Reading '$p'");
if($verbose) {
$printer->output_permanent($cmd);
}
return map { s!^\Q$p\E[/\\]?!!r }
split /\r?\n/,readpipe( $cmd );
} else {
opendir my $dh, $dir
or die "Can't read '$dir': $!";
return readdir($dh);
}
}
This allows me to keep all filtering local in perl, which is not efficient over the network but highly efficient in protecting me from needing to shell-escape all the commands.
Naming the image import directories
I often end up with directories named 20260417-1603 , which is better than
nothing. But I still would like to know what happened then. Often I have an
entry in my calendar which might be so-and-sos birthday party, or some
conference or whatnot, and having the computer automatically naming the
directory according to the calendar is a nice touch. I've put off this for a
long time, but now I've actually released
Net::CalDAV::FindEntry
to make this convenient:
my @relevant_events = $cal->get_events(
after => $image_timestamp,
before => $image_timestamp,
calendar => 'Corion'
);
# We blindly take the event that started closest before the photo was taken
my ($ev) = sort { $a->{start} cmp $b->{start} } @relevant_events;
if( $ev ) {
$target_directory{ $image } //= "$image_timestamp - $ev->{title}";
}
With that line added, I get conveniently pre-named directory names on the central server. It's still not perfect, as the directory naming needs to respect timezones while the image timestamps are either in a floating timezone or GMT, but this gets me a long step forward in convenience. Also, I need to clean up the titles of the calendar entries to make them suitable as directory names. Originally I thought Text::CleanFragment would be good for this, but it replaces spaces with underscores, which I do not want in the directory names, so I wrote a small ad-hoc sanitizer for that. So there still is work to do in the future.
