One of my yearly things is to post-process and publish the videos
we (well, Lee Johnson) record of the German Perl Workshop.
The videos get recorded in OBS Studio and already get the
sidebar with the sponsor information, talk and speaker information. After the workshop has
concluded, these files then need to be post-processed.
Most of it is automated with a large Makefile. This covers
Importing the talk metadata from Act
Setting up a cutting pipeline to trim the talks
Adding the metadata of speaker and talk title to each video
Adding a header and a trailer to each talk
My main tasks there are to create an SVG image to use as a titlecard, and to wrestle with ffmpeg
so it uses hardware acceleration instead of only CPU processing.
After producing the final videos, each talk gets reviewed for obvious bad stuff, like audio cutting out
or otherwise too bad quality.
The last step then is to upload the videos to Youtube, create a new playlist there and retitle
all the uploaded videos with the event, speaker and title. This is mostly a manual task, since automating
Youtube would mean to have to deal with another always-shifting API.
After all these manual steps, I'm still happy to present the GPW2026 videos
to you.
Now that I have a shell script to send images to my Eink display, I'm looking
at improving the experience. Mostly by porting the tools to Perl to better
integrate them with my other tools. But also, by making it more convenient to
find and select new images to display on the screen.
Shell script
I want to build a good collection of images to display on the Eink display.
As I often lose track of the filename of a very good looking image (
I'm already looking at it on the display and I like it, why should I bother with the filename?
), I've added logging to my interim shell script so that I have a log of files
to go through and re-evaluate.
I'm using JSONL as the log format, not because it is especially easy to create
from the shell, but it is highly convenient for later processing with other
tools.
#!/usr/bin/perl
use 5.020;
use JSON::Tiny 'encode_json';
use Getopt::Long (qw(:config pass_through permute)); # stop at the first unknown option or first argument
use POSIX 'strftime';
my $result = {};
my $name;
GetOptions(
'<>' => sub( $v ) { if( ! $name ) { $name = $v =~ s/^--//r; } else { $result->{$name} = $v; undef $name }; },
);
$result->{ timestamp } //= strftime '%Y-%m-%dT%H:%M:%SZ', gmtime();
say encode_json( $result );
I've also added bas64-encoded thumbnail images to the log lines so that I can
easily display the logfile as an HTML page to better pick out good images or
images to discard.
Shell integration
After adding the appropriate file in ~/.local/share/applications/eink.desktop
I can send any image to the frame by using a right-click. This is highly
convenient and encourages me to quickly try out new images.
The Perl port of the sender script will likely use
request/response mechanism, easily adapted into Future / Future::Mojo
I need to port the mdc Javascript module and create a test suite to properly
encode the blobs.
MDC documentation PDF
While looking at the Javascript code, I found the official MDC documentation
version 15.0 PDF from 2020-11-06 . This describes the actual protocol
(serial, or over IP) in detail.
The command blocks are simple, consisting only of 8-bit bytes. This is a
problem for payloads (like URLs) that are longer than 230 bytes. But for my
use-case, that is no problem.
The inconvenient thing is that the length does not immediately precede the
payload, so that unpack does not work. Instead I've used a regular expression
to extract the parts.
The blobs are sent and received via port 1515. I have not found a way to
change this port, but I'm not sure why I would want that.
This post is part of the notes I took while preparing for and during the migration of https://perlmonks.org behind a CDN. I mostly publish these notes so that I can find them again later.
There are situations where I don't want to override the DNS resolution for my complete system, but still connect to specific machines pretending a DNS name resolves to them. Examples are SSL certificate checking, and various firewall configurations. Especially while migrating a site behind a CDN or when setting up s site beh[nd a reverse proxy or Wireguard, I want to inspect and compare the results of queries to the different machines.
For these examples, assume that 10.0.0.1 is the machine serving the website
perlmonks.org as the origin. The public DNS resolves to a pool of CDN machines
somewhere else, but we want to debug what the original source is serving.
The original machine also wants to be accessed as perlmonks.org over SSL.
Open the Firefox browser console with Ctrl+Shift+J . You should be able to enter Javascript there; If not, enable "Debugging Tools für Browser-Chrome" in the normal browser console settings (F12 , then F1yeah, CUI standards be damned ). Then enter the name/IP pair for name resolution. This persists until you restart Firefox.
For LWP::UserAgent there is no general mechanism, but monkeypatching LWP::Protocol::https works. To also make SNI work, you
should additionally pass in the SSL_hostname explicitly to the SSL options. I'm not sure why this is necessary, as the code in LWP::Protocol::https
extracts the hostname from the request URL, but this made the difference for me between 421 Misdirected Request and 200 OK with Fastly :
our $force_peeraddr;
around 'LWP::Protocol::http::_extra_sock_opts' => sub {
my $orig = shift;
die unless wantarray;
my @rv = $orig->(@_);
push @rv, PeerAddr => $force_peeraddr if defined $force_peeraddr;
return @rv;
};
around 'LWP::Protocol::https::_get_sock_info' => sub {
my $orig = shift;
my ($self, $res, $sock) = @_;
my $cert = $sock->get_peer_certificate;
my @san = $cert->peer_certificate('subjectAltNames');
use Data::Dumper; warn Dumper \@san;
while (@san) {
my ($type_id, $value) = splice @san, 0, 2;
$res->push_header("Client-SSL-Cert-SubjectAltName"
=> "$type_id: $value");
}
$orig->(@_);
};
$force_peeraddr = '10.0.0.1';
my $ua = LWP::UserAgent->new(
ssl_opts => {
verify_hostname => 0,
SSL_hostname => 'perlmonks.org', # for SNI
},
timeout => 60,
);
I really like live-editing documents. Interactively seeing the results of your edits makes editing more fun. For tools made by others, this would mean WYSIWYG, but for tools made by (and for) myself, I prefer the approach
of editing a text file and having the conversion process kick off automatically on every file save.
For some reason, I mostly prefer editing text in a plain/small editor as ASCII. My markup needs are usually restricted to bold and italics, with a sprinkling of images and links. I don't have particular layout needs - most of that is covered by a template.
The main idea is to have two programs, one program for editing (commonly, a text editor) and one program for converting and displaying the rendered file. Instead of having a program that does the display and editing for
one or more file formats, this separation allows me to use the convenient text editor that I've customized to my liking instead of a built-in text editor. It also
allows me to add arbitrary intermediate steps to convert from the source to the display.
This idea is nothing new - the Emacs Flymake mode
does recompilation (and other recreation) in the background whenever a source file changes as well.
This program runs make whenever a file in a given directory or its subdirectories changes. The Makefile
should contain the rules to regenerate the interesting content. The default rules
are only for converting LaTeX files into PDF:
# This is the default Makefile to be used by the asset reloader
# if no Makefile is found in the current directory
OPEN=xdg-open
MAYBE_OPEN=perl /home/corion/Projekte/App-LiveEdit/scripts/maybe-open.pl
.PHONY: all
all: .pdf
.pdf.tex:
pdflatex -interaction=batchmode "$<"
$(MAYBE_OPEN) "$@"
make solves the problem backwards, given a file that I want, regenerate everything
intermediate. live-edit.pl solves the problem forwards, given a file that changed, regenerate everything that depends on it.
Maybe live-edit.pl should be named ekam instead.
maybe-open.pl - this is an offshoot program that opens a file using xdg-open
if no process
is running with the file name on its command line. Highly convenient for launching
a .pdf file once in its viewer.
It's weird but currently I draw most of my programming satisfaction from using
(and enhancing) App::sqldisplay.
It is a simplistic app that allows me to write and display SQL queries against
a spreadsheet.
I mostly use it to do the accounting for the Perl club. But doing that made
doing the books much more fun actually.
Here I edit a cell in the spreadsheet, and the SQL query in the browser
automatically updates and highlights the changed rows:
Also, I've now started exploring
HTMX
for that, and it works surprisingly well for the use case I have, removing
20+ lines of code in favour of 1 line of configuration and small Perl-side
code changes.
The one thing I don't like about HTMX is that for every component of your page
that you want to update individually, you need to create an HTTP accessible
endpoint. But for push notifications, you can just push the new HTML instead,
which is good enough.
But "even" with this hobby project, I see feature creep, as just today I thought
about exporting the collection of queries as an Excel sheet. And the UI now has
tabs, just like an Excel sheet and I wonder if I should keep an Excel
sheet/spreadsheet for data entry or just (also) write my own editing UI. But
writing an editing UI is makework for little gain. I would
want it to have mostly Excel-like UI, keyboard bindings and autofill.
