Practicing Pragmatism

While using query-replace I entered into the recursive edit mode – I've struggled to use this in the past for any great effect.. – so its no wonder I managed to /leave/ this mode on.

Later on during my session I tried to start up an emacsclient (with -nw to use the tty in a terminal emulator) and found that Emacs didn't want to respond to any keypress :(

Well. What do you do? You turn to a debugging tool of course! In this case, I have the handy M-x doom/toggle-profiler command to use from the excellent doom-emacs framework. This command toggles the CPU and memory profilers that Emacs has built in to the “on” position and records away. So, after starting the profiler all I had to do was start up my emacsclient client again and see what the heck was going on.

In my head I thought it could be:

  1. An issue with my unusual invocation (fzf | xargs --open-tty --no-run-if-empty -t -- emacsclient -nw)
  2. The main loop was stuck on a piece of code
  3. $TERM, Emacs, and the terminal's tcinfo were in disagreement and could be spewing unknown/unhandled control characters around

Without digging into things to hard I found that the recursive edit was still in progress.. because it was the parent frame for every other frame recorded:

Note that the top frames are from first starting query-replace!

Neat.

Exiting the recursive edit fixed the locked sessions and everything is working much better now :)

Can't say its a bug for sure – yet – because I'd need to try coming at this from the repro case to make a helpful report. Which I just might do.

I wanted to get home-manager installed on a system of mine. Most of them have been set up to support the use of flakes and so far... its caused a lot of headaches in practice & operation. Don't read this wrong way – I'm happy with the results, but still learning the new pathways.

Anyhow, I wanted to do the equivalent of nix-env -iA nixpkgs.hello with a flake. None of the documentation seems to make any suggestion nor do they actually support Nix Flakes directly (nix-env is a very very old tool in the Nix world, along with concepts like nix-channel!). So, to install something from a flake, I got creative:

# Add entry to registry for aliased name (home-manager):
nix registry add home-manager github:nix-community/home-manager
# Build and add the default package to the user profile:
nix build 'home-manager#' --profile $HOME/.nix-profile
# Observe!
command -v home-manager
#=> /home/jake/.nix-profile/bin/home-manager

It's there!

Whether or not features in this particular tool (home-manager) supports running under this configuration is a different story.

I'll find out either way.

Here's a problem I was trying to resolve for myself. Being me, a engineer by trade who designs robust and future thinking solutions (so I can be safely lazy later), I have a tendency to over design, over think, and over build for the present scope of need. I've delivered on many of these designs by splitting areas of high return from features & nice-to-haves.. yadayada. Point is that at some point I catch myself before I lose the carefully stacked idea in my mind and make sure I get it out there!

In this case: I had set up a cluster of nodes running Hashicorp's Vault that I wanted to use to vend dynamic credentials for InfluxDB, MQTT, OpenID/JWT Authentication & Verification, Secrets storage, and various KMS functions.

These nodes provide their secrets just fine.. which is good? I promise they're only telling secrets to the right folks. To this very aim I found myself becoming more and more annoyed at the thought that I'd have to write yet another configuration template that I would have to deploy alongside my applications. Instead, I thought: huh, we have this fancy new embed package in Go and it probably could be used to make this a bit better – that'd be neat to be able to use Vault's rich Go clients (and many many other Go based behemoths.. ahem, k8s).

Ideas

I have a handful of approaches I want to take here.

  1. Provide alternative entrypoint in ELF executable that hits an appended software entrypoint that's responsible for fetching credentials prior to the application starting up. The formatted credentials inserted applicable argv and environment variables by the appended entrypoint then configure the desired end application with runtime fetched secrets and configuration values (the process replaced by exec, or by literally pointing to its address and starting over? I have no idea how to make ELF magic work).

  2. Provide in-memory file descriptors to closed files inherited by a child process

  3. Hook into the native code with wrapping credentials fetches (the “we have the source Luke!” idea where I stop getting fancy). This assumes that the application itself provide an interface to “run it”, like: PlainMain(context.Context, string, []string) error.

  4. (3a) Hook into the native code by sharing entries in the embedded filesystem tree! No idea whether or not this is even possible to do. I learned about the added package just after 1.16 was released. Either way, I'll try this out too.

  5. (3b) Call embedded memory blobs from the embed filesystem and set memory locations with credential bearing cstrings (probably not much better than #!/usr/bin/env sh\n\nexec ./silly -pw Hunter1, really).

I have a very old Xbox One with aging metal hurtling around to fetch games, which gives me and my partner a chance to grab coffee, maybe take the dog out, and have 30 seconds to spare before the next loading screen. Our Xbox came with a Kinect and Sunset Overdrive – Day One Edition – a fantastic game. I really mean it when I say it's old.

Recently though, we've gotten back to playing through some uncompleted games and they load slower than molasses. Final Fantasy XV? Yeah, go out and get a coffee, a gluten free bagel, and then come back, eat, and get ready to play!!

Waiting a long time between loads, whether due to Game Overs or not, is not fun.

So? What can we do? Replace the spinning platter(s) of metal in the console and move on to snappy solid state storage!

That's what exactly what I did.

  1. Uncase the Xbox (while its fully disconnected, naturally)
  2. Remove the spinning metal disk
  3. Plug in the spinning metal disk into a machine along with the new image
  4. Use dd to copy the image from the original disk to the, ideally, equally sized or larger, solid state disk that'll go back in its place. I've seen it written that there's limitations on the actual disk size supported, I went with a 2.5” 1TB Samsung 860 EVO to replace the 2.5” 5200 RPM Seagate laptop hard drive that was in there and had no issue.
  5. After dd-ing, attach the disk to a Window's “box” (I used a handy VM that I passed the block device into).
  6. With the Windows host, use diskmgr to resize the largest existing NTFS partition to occupy any newly available space (on your new disk, if its bigger than the original).
  7. Remove from the Windows machine (unmount, detach, etc.) and extract the disk to be inserted into the Xbox
  8. Insert the disk and reassemble the Xbox case with the disk secured where the old spinning hard drive was.
  9. Power back on and skip the trip to the coffee shop, because you're playing on a console with solid state storage! I think we saw a minute and a half (90+ seconds) drop to about 30 seconds to load Final Fantasy XV, I didn't take down good notes of the time, sorry.

I had hoped this post to be more “full”, but life got in the way and had to move with short notice winter 2020 (just before the holidays here in the US). If there's details folks would like to see added, please reach out to me – I'd be happy to add more in that case: editor AT prag.dev

Warning: this is more or a less a #rant about #docker & #containers (and is of my own strong opinions about containers).

It shouldn't matter whether or not docker is dying, dead, or otherwise. Some folks seem to be elated that Docker will no longer be supported with Kubernetes, other folks seem confused. And there's others, like myself, who see Docker as having built something incredible and having stymied itself trying to transform into a business.

The way I see it, docker's introduction fundamentally changed the way the software, and its infrastructure sub-industry, operate. However, the business side of things made me think that there must have been a significant internal imbalance where idealistic technologists, engineers, and those marketing their efforts were fully at odds. Luckily for us moby (moby having been renamed from Docker to disassociate tech from the business side of things) spawned what is now the Open Containers Initiative and its many specifications for container technologies.

Docker isn't a fundamental player any longer. We have Docker to thank, praise, and continue to use & appreciate for their contributions to Linux primitives – cgroups, psi, some systemd developments, and more. Docker is a fantastic development tool and, despite growing beyond their core components, continues to innovate in building out tools for the community – buildkit, linuxkit, docker-compose (remember when that was python?).

Personally, I have to thank dockerd and its roots for pulling me into its “containers” orbit, teaching me Go, and offering a platform on which to build out my own ideas.

I'm excited for what Docker can become having been liberated from Kubelet. They've done turnabouts before, and I hope this time Docker improves its uses of containerd, runc, and other CRI runtimes while leaving aside related, but distracting efforts. Sheer unbridled intuition tells me that we're going to see them either innovate, thriving on the shoulders of foundational tech.. or fall flat with the community making moby the tool they want and need it to become.

As always, this is my opinion and does not reflect anyone else's or any of my employer's perspective on these matters.

Over the years I have appreciated docker, understood its purpose, exercised it to great results, and hope only to it morph into its next phase in our collective engineering toolboxes. Kubernetes doesn't need Docker and Docker doesn't need to be the bedrock (or in the critical path, as it were) of Kubernetes. Being independent, a tool once again safely malleable, makes for interesting opportunities.

NixOS is a favorite of mine. I waste little opportunity to mention it when coworkers lament or when ideas come up that are not only deftly solved by Nix and solved in a way that's remarkably similar to their own lofty goals or vision. I'm not concerned with even promising more posts about #Nix, #NixOS, and Nixpkgs – they're coming. Whether I want them to or not. Be prepared.

NixOS is a favorite, but it's also quirky as hell.

The #linux distribution isn't your typical one: it has everything content addressed, wrapped to deal with this, and is immutable in nature (ostensibly so). With the advantages (more to come, or read the NixOS website) that it brings, it also brings along its deficiencies and disadvantages. In my case, and for the focus of this post, this recently bit me when trying to make a proprietary driver work for the video capture card that I bought: a Blackmagic Design DeckLink 4K Mini Recorder (whoa, that's a stupidly long name).

The driver from the card's company is distributed behind a “click to accept the terms of this license” button and, more disappointingly, with proprietary binary blobs. Yay.

For most [^1] Linux users, this isn't an issue. Companies that “support Linux users” will usually [^1] build their blobs with the same toolchain and shared libraries used by popular distributions. When they don't, I've seen them drop in their own shared libraries and plan on the user having at least a compatible linker and kernel ABI [^2]. There's a fun story here about some ancient IBM TTS system.. for another time. In my case, NixOS being a not-popular nor “typical” distribution, I had issues.

NixOS' kernel sources don't land in /usr/src. It's kernel modules don't land in /lib/modules/$(uname -r). They're in /run/current-system/kernel-modules. Ugh. The differences don't end there and they're nuanced. If a linker was hardcoded into the executable, proprietary binary blobs.. well, on my system those paths are wrong. I don't install the .deb, or the .rpm. Typically, I install software my distribution packages with commands like nix-env -iA nixos.ripgrep. These commands wind up pulling in runtime dependencies (or build dependencies for a fallback build) and place the “package” in /nix/store. This works very well for appropriately licensed and distributed software – but the closed source ones.. they take the cake for adventures to places I didn't contemplate going.

These drivers, admittedly, were not closed source. So? Was it easy? Not a walk in the park, but not too bad. I've dealt with worse on NixOS.

The Linux solution published by the fine folks at Blackmagic Design, included a few utilities, helper executables, and, of course, the kernel module for the PCIe card itself. There's a separate SDK vended too – it's needed for building ffmpeg with decklink support, but that's very out of scope.

Back to the kernel module.

This expression handles a couple things:

  • unpack the assumptive tarball (it's basically an rpm but not)
  • patching the upstream sources (to actually work.. thanks to Arch Linux maintainers here!)
  • building the kernel module (it's a more-or-less standard process)
  • ripping out “impure” [^3] or unneeded references (using nuke-refs)

I'm not explaining Nix expressions in this go, maybe another time. I'm continuing on through anyway.

{ stdenv
, fetchpatch
, nukeReferences
, linuxPackages
, kernel ? linuxPackages.kernel
, version
, src
}:

stdenv.mkDerivation {
  name = "blackmagic-${version}-module-${kernel.modDirVersion}";
  inherit version;

  buildInputs = [ nukeReferences ];

  kernel = kernel.dev;
  kernelVersion = kernel.modDirVersion;

  inherit src;
  patches = [
    (fetchpatch {
      name = "fix-get_user_pages-and-mmap_lock.patch";
      url = "https://aur.archlinux.org/cgit/aur.git/plain/02-fix-get_user_pages-and-mmap_lock.patch?h=decklink&id=8f19ef584c0603105415160d2ba4e8dfa47495ce";
      sha256 = "08m4qwrk0vg8rix59y591bjih95d2wp6bmm1p37nyfvhi2n9jw2m";
    })
    (fetchpatch {
      name = "fix-have_unlocked_ioctl.patch";
      url = "https://aur.archlinux.org/cgit/aur.git/plain/03-fix-have_unlocked_ioctl.patch?h=decklink&id=8f19ef584c0603105415160d2ba4e8dfa47495ce";
      sha256 = "0j9p62qa4mc6ir2v4fzrdapdrvi1dabrjrx1c295pwa3vmsi1x4f";
    })
  ];

  postUnpack = ''
    cd */usr/src
    sourceRoot="$(pwd -P)"
  '';

  buildPhase = ''
    cd $sourceRoot/blackmagic-''${version}*/
    # missing some "touch" commands, make sure they exist for build.
    touch .bmd-support.o.cmd
    make -C $kernel/lib/modules/$kernelVersion/build modules "M=$(pwd -P)"

    cd $sourceRoot/blackmagic-io-''${version}*/
    # missing some "touch" commands, make sure they exist for build.
    touch .blackmagic.o.cmd
    make -C $kernel/lib/modules/$kernelVersion/build modules "M=$(pwd -P)"

    cd $sourceRoot
  '';

  installPhase = ''
    mkdir -p $out/lib/modules/$kernelVersion/misc
    for x in $(find . -name '*.ko'); do
      nuke-refs $x
      cp $x $out/lib/modules/$kernelVersion/misc/
    done
  '';

  meta.platforms = [ "x86_64-linux" ];
}

Building the kernel module was a piece of cake, really. The meat of the “interesting parts” here was a snippet I kept from another round of kernel module hackery (oh, FireWire..):

    make -C $kernel/lib/modules/$kernelVersion/build modules "M=$(pwd -P)"

The above line builds the appropriate kernel module with, and for, the provided kernel. That's pretty typical of any given kernel module – in tree and out of tree modules. When the above derivation is realised (ie: built), the derivation (ie: a package, but not in the traditional sense) outputs a kernel module – the .ko – such that it'll appear at /run/current-system/kernel-modules/lib/modules/5.9.10/misc. modprobe is configured on NixOS hosts to use this directory, so all is well!

But, that's not all. I mentioned that there were helper executables. They're needed in order to setup a capture device when the PCI probe matches one by way of udev. Without the helpers, the device is halfway initialized – software that's capable of using the correct IOCTLs can't even see the devices until the helpers have a chat with the kernel module.

In full glory, here's the tools. Just the tools. There's still the SDK too..

{ stdenv
, autoPatchelfHook
, makeWrapper
, libGL, libGLU
, libuuid
, dbus_libs
, alsaLib
, xorg
, fontconfig
, freetype
, glib
, version
, src
, mediaexpress ? null
}:

stdenv.mkDerivation {
  pname = "blackmagic-tools";
  inherit version;

  inherit src mediaexpress;

  donStrip = true;

  nativeBuildInputs = [ makeWrapper autoPatchelfHook ];
  buildInputs = [ libGL libGLU libuuid dbus_libs freetype fontconfig glib alsaLib ]
                ++ (with xorg; [libxcb libXrender libICE libX11 libXinerama libXrandr libSM ]);


  postUnpack = ''
    if [[ -s "$mediaexpress" ]]; then
      tar -C $sourceRoot --strip-components=1 -xf $mediaexpress
    fi
  '';

  doBuild = false;

  installPhase = ''
    bins=( $(cd usr/bin; ls) )
    # Add the helpers to bin. They're needed by udev rules and the supporting
    # systemd service.
    bins+=( DesktopVideoNotifier DesktopVideoHelper )
    # Prune vended systemd units and the symlinked bin stubs.
    rm -rf usr/lib/systemd usr/bin

    cp -R usr $out

    # Replace symlinks with wrappers to include dynamicly dlopen'd libraries.
    # patchelf may corrupt the executables when adding a static entry that would
    # normally influence the RPATH.
    libBin=$out/lib/blackmagic/DesktopVideo
    for x in "''${bins[@]}"; do
      towrap="$out/lib/blackmagic/MediaExpress/$x"
      if ! [[ -x "$towrap" ]]; then
        towrap="$libBin/$x"
      fi
      makeWrapper $towrap $out/bin/$x \
        --prefix LD_LIBRARY_PATH ':' $out/lib \
        --prefix QT_PLUGIN_PATH ':' $libBin/plugins
    done

    # Need to substitute the executable path in these rules to use the $out/bin
    # path.
    mkdir -p $out/lib/udev/rules.d
    substitute \
      etc/udev/rules.d/55-blackmagic.rules \
      $out/lib/udev/rules.d/55-blackmagic.rules \
      --replace /usr/lib/blackmagic/DesktopVideo/DesktopVideoNotifier \
               $out/lib/blackmagic/DesktopVideo/DesktopVideoNotifier
  '';

  preFixup = ''
    # add $out/lib to the RPATH set on executables to use bundled version of
    # Qt5.
    runtimeDependencies+=" $out"
  '';
}

The tools provided the helpful little viewer and card setting utilities, right next to DesktopVideoNotifier and DesktopVideoHelper which provide the other half of capture card initialization. So, because I was being pragmatic, I lumped together these tools and did “brain surgery” in the context of executables by using patchelf (by way of autoPatchelf): the bundled shared libraries and expected runtime libraries, which were expected to be present like the popular crowds, were all in fact shoehorned into the executables' RPATH. And then wrapped.

As it turns out, wrappers are a good thing. Nixpkgs uses them liberally to improve the flexibility and reusability of its more build-time-consuming derivations (not as a goal per-se). This saves on build time and also disk space – if you wrap the thing you want to customize in a way that permits stubbing or subbing out dependencies, runtime configuration, or plain 'ol dependency resolution then you can easily change your mind later on. If you read through packages in the Nixpkgs repository, you'll come to see that its not uncommon to have packages several layers deep of derivations. Often times they're layered specifically to allow users to override some characteristic or improve composition of multiple derivations to produce a final useful environment.

You might have noticed there's missing.. everything.. that should have connected these up to be something. In this case, you'll have to deal with it. I returned the capture card after struggling to keep it and the devices plugged into it on the correct output/input formats – on top of the whole “we're NVIDIA and what is a GPL?” thing here in November of 2020 which made CUDA/NVENC workloads non-functional with Linux 5.9.

I've got the code hanging around and arguably should flesh this out more. I used more words than planned anyhow, so this'll be it.

[^1]: I'm not going to even try to quantify

[^2]: This is cool stuff, there's lot's to read on this topic, foo

[^3]: Just search for impure in the Nix manual and Nixpkg's, purity is a nuisance to the solution that Nix proposes.