Why?

Recently I took up photography again. Or rather, I made more of a conscious effort to take pictures again, as opposed to snapping them. I have my existing image import pipeline, which copies the files from the devices to my central server into timestamped directories.

Recently I actually exhausted the storage of my mobile phone and that made me realize that I want to run my photo import via rsync from abroad on my phone instead of local-only copying via mounted drives.

Reading files

Before the change, the program took all files in the DCIM directory and copied them via an SMB mount to the central server:

• Find all files in DCIM/Camera directories
• Read the image creation times from image EXIF data
• Upload all files into directories grouped by their timestamps
• Move all files into the DCIM/Camera/archive as a safety measure. I manually delete the archive/ directory from time to time

But running the import on a gphoto2 mounted directory does not allow to move transferred files to the DCIM/archive directory. This would be ameliorated by using rsync. but a more discerning approach is still needed since I might have renamed the target directories on the central server in the meantime.

The new approach addresses these points

• Find all files in DCIM/Camera directories
• Find (rough) creation date from image filename (fast), or from image EXIF data (slow, needed for sequentially numbered image file names)
• Read the remote directories starting from the earliest image timestamp
• Associate each image with a target directory
  • if it already exists on the remote side, use that directory
  • use the image timestamp for the image otherwise
• Upload all files into their target directories using rsync

Using rsync and always transferring / synchronizing all images makes the transfer resilient against network failures and partial transfers. Looking at the already existing filenames on the target end also opens the door to reading calendar entries and creating the directories with their calendar entry outright, as I've decoupled the directory name from the image creation timestamp.

Having an ssh remote target

Moving to rsync also enables backing up to a remote machine over ssh. To move the files into the appropriate locations, I also need to read remote directories. I did not find an established module on CPAN for this. Filesys::Virtual::SSH is OKish, but I don't like the API of Filesys::Virtual overall. I only needed a single routine to read a remote directory, so I wrote a very simple wrapper:

sub r_readdir($dir, $type="d") {
    if( $dir =~ m!^ssh:(?<host>(\w+\@)?[^:]+):(?<path>.*)! ) {
        my $p = $-{path}->[0];

        my $cmd = "ssh '$-{host}->[0]' 'find \"$p\" -type $type'";

        currently("Reading '$p'");
        if($verbose) {
            $printer->output_permanent($cmd);
        }

        return map { s!^\Q$p\E[/\\]?!!r }
            split /\r?\n/,readpipe( $cmd );
    } else {
        opendir my $dh, $dir
            or die "Can't read '$dir': $!";
        return readdir($dh);
    }
}

This allows me to keep all filtering local in perl, which is not efficient over the network but highly efficient in protecting me from needing to shell-escape all the commands.

Naming the image import directories

I often end up with directories named 20260417-1603 , which is better than nothing. But I still would like to know what happened then. Often I have an entry in my calendar which might be so-and-sos birthday party, or some conference or whatnot, and having the computer automatically naming the directory according to the calendar is a nice touch. I've put off this for a long time, but now I've actually released Net::CalDAV::FindEntry to make this convenient:

my @relevant_events = $cal->get_events(
    after    => $image_timestamp,
    before   => $image_timestamp,
    calendar => 'Corion'
);

# We blindly take the event that started closest before the photo was taken
my ($ev) = sort { $a->{start} cmp $b->{start} } @relevant_events;
if( $ev ) {
  $target_directory{ $image } //= "$image_timestamp - $ev->{title}";
}

With that line added, I get conveniently pre-named directory names on the central server. It's still not perfect, as the directory naming needs to respect timezones while the image timestamps are either in a floating timezone or GMT, but this gets me a long step forward in convenience. Also, I need to clean up the titles of the calendar entries to make them suitable as directory names. Originally I thought Text::CleanFragment would be good for this, but it replaces spaces with underscores, which I do not want in the directory names, so I wrote a small ad-hoc sanitizer for that. So there still is work to do in the future.