Practicing Pragmatism

Here's a problem I was trying to resolve for myself. Being me, a engineer by trade who designs robust and future thinking solutions (so I can be safely lazy later), I have a tendency to over design, over think, and over build for the present scope of need. I've delivered on many of these designs by splitting areas of high return from features & nice-to-haves.. yadayada. Point is that at some point I catch myself before I lose the carefully stacked idea in my mind and make sure I get it out there!

In this case: I had set up a cluster of nodes running Hashicorp's Vault that I wanted to use to vend dynamic credentials for InfluxDB, MQTT, OpenID/JWT Authentication & Verification, Secrets storage, and various KMS functions.

These nodes provide their secrets just fine.. which is good? I promise they're only telling secrets to the right folks. To this very aim I found myself becoming more and more annoyed at the thought that I'd have to write yet another configuration template that I would have to deploy alongside my applications. Instead, I thought: huh, we have this fancy new embed package in Go and it probably could be used to make this a bit better – that'd be neat to be able to use Vault's rich Go clients (and many many other Go based behemoths.. ahem, k8s).

Ideas

I have a handful of approaches I want to take here.

  1. Provide alternative entrypoint in ELF executable that hits an appended software entrypoint that's responsible for fetching credentials prior to the application starting up. The formatted credentials inserted applicable argv and environment variables by the appended entrypoint then configure the desired end application with runtime fetched secrets and configuration values (the process replaced by exec, or by literally pointing to its address and starting over? I have no idea how to make ELF magic work).

  2. Provide in-memory file descriptors to closed files inherited by a child process

  3. Hook into the native code with wrapping credentials fetches (the “we have the source Luke!” idea where I stop getting fancy). This assumes that the application itself provide an interface to “run it”, like: PlainMain(context.Context, string, []string) error.

  4. (3a) Hook into the native code by sharing entries in the embedded filesystem tree! No idea whether or not this is even possible to do. I learned about the added package just after 1.16 was released. Either way, I'll try this out too.

  5. (3b) Call embedded memory blobs from the embed filesystem and set memory locations with credential bearing cstrings (probably not much better than #!/usr/bin/env sh\n\nexec ./silly -pw Hunter1, really).

I have a very old Xbox One with aging metal hurtling around to fetch games, which gives me and my partner a chance to grab coffee, maybe take the dog out, and have 30 seconds to spare before the next loading screen. Our Xbox came with a Kinect and Sunset Overdrive – Day One Edition – a fantastic game. I really mean it when I say it's old.

Recently though, we've gotten back to playing through some uncompleted games and they load slower than molasses. Final Fantasy XV? Yeah, go out and get a coffee, a gluten free bagel, and then come back, eat, and get ready to play!!

Waiting a long time between loads, whether due to Game Overs or not, is not fun.

So? What can we do? Replace the spinning platter(s) of metal in the console and move on to snappy solid state storage!

That's what exactly what I did.

  1. Uncase the Xbox (while its fully disconnected, naturally)
  2. Remove the spinning metal disk
  3. Plug in the spinning metal disk into a machine along with the new image
  4. Use dd to copy the image from the original disk to the, ideally, equally sized or larger, solid state disk that'll go back in its place. I've seen it written that there's limitations on the actual disk size supported, I went with a 2.5” 1TB Samsung 860 EVO to replace the 2.5” 5200 RPM Seagate laptop hard drive that was in there and had no issue.
  5. After dd-ing, attach the disk to a Window's “box” (I used a handy VM that I passed the block device into).
  6. With the Windows host, use diskmgr to resize the largest existing NTFS partition to occupy any newly available space (on your new disk, if its bigger than the original).
  7. Remove from the Windows machine (unmount, detach, etc.) and extract the disk to be inserted into the Xbox
  8. Insert the disk and reassemble the Xbox case with the disk secured where the old spinning hard drive was.
  9. Power back on and skip the trip to the coffee shop, because you're playing on a console with solid state storage! I think we saw a minute and a half (90+ seconds) drop to about 30 seconds to load Final Fantasy XV, I didn't take down good notes of the time, sorry.

I had hoped this post to be more “full”, but life got in the way and had to move with short notice winter 2020 (just before the holidays here in the US). If there's details folks would like to see added, please reach out to me – I'd be happy to add more in that case: editor AT prag.dev

Warning: this is more or a less a #rant about #docker & #containers (and is of my own strong opinions about containers).

It shouldn't matter whether or not docker is dying, dead, or otherwise. Some folks seem to be elated that Docker will no longer be supported with Kubernetes, other folks seem confused. And there's others, like myself, who see Docker as having built something incredible and having stymied itself trying to transform into a business.

The way I see it, docker's introduction fundamentally changed the way the software, and its infrastructure sub-industry, operate. However, the business side of things made me think that there must have been a significant internal imbalance where idealistic technologists, engineers, and those marketing their efforts were fully at odds. Luckily for us moby (moby having been renamed from Docker to disassociate tech from the business side of things) spawned what is now the Open Containers Initiative and its many specifications for container technologies.

Docker isn't a fundamental player any longer. We have Docker to thank, praise, and continue to use & appreciate for their contributions to Linux primitives – cgroups, psi, some systemd developments, and more. Docker is a fantastic development tool and, despite growing beyond their core components, continues to innovate in building out tools for the community – buildkit, linuxkit, docker-compose (remember when that was python?).

Personally, I have to thank dockerd and its roots for pulling me into its “containers” orbit, teaching me Go, and offering a platform on which to build out my own ideas.

I'm excited for what Docker can become having been liberated from Kubelet. They've done turnabouts before, and I hope this time Docker improves its uses of containerd, runc, and other CRI runtimes while leaving aside related, but distracting efforts. Sheer unbridled intuition tells me that we're going to see them either innovate, thriving on the shoulders of foundational tech.. or fall flat with the community making moby the tool they want and need it to become.

As always, this is my opinion and does not reflect anyone else's or any of my employer's perspective on these matters.

Over the years I have appreciated docker, understood its purpose, exercised it to great results, and hope only to it morph into its next phase in our collective engineering toolboxes. Kubernetes doesn't need Docker and Docker doesn't need to be the bedrock (or in the critical path, as it were) of Kubernetes. Being independent, a tool once again safely malleable, makes for interesting opportunities.

NixOS is a favorite of mine. I waste little opportunity to mention it when coworkers lament or when ideas come up that are not only deftly solved by Nix and solved in a way that's remarkably similar to their own lofty goals or vision. I'm not concerned with even promising more posts about #Nix, #NixOS, and Nixpkgs – they're coming. Whether I want them to or not. Be prepared.

NixOS is a favorite, but it's also quirky as hell.

The #linux distribution isn't your typical one: it has everything content addressed, wrapped to deal with this, and is immutable in nature (ostensibly so). With the advantages (more to come, or read the NixOS website) that it brings, it also brings along its deficiencies and disadvantages. In my case, and for the focus of this post, this recently bit me when trying to make a proprietary driver work for the video capture card that I bought: a Blackmagic Design DeckLink 4K Mini Recorder (whoa, that's a stupidly long name).

The driver from the card's company is distributed behind a “click to accept the terms of this license” button and, more disappointingly, with proprietary binary blobs. Yay.

For most [^1] Linux users, this isn't an issue. Companies that “support Linux users” will usually [^1] build their blobs with the same toolchain and shared libraries used by popular distributions. When they don't, I've seen them drop in their own shared libraries and plan on the user having at least a compatible linker and kernel ABI [^2]. There's a fun story here about some ancient IBM TTS system.. for another time. In my case, NixOS being a not-popular nor “typical” distribution, I had issues.

NixOS' kernel sources don't land in /usr/src. It's kernel modules don't land in /lib/modules/$(uname -r). They're in /run/current-system/kernel-modules. Ugh. The differences don't end there and they're nuanced. If a linker was hardcoded into the executable, proprietary binary blobs.. well, on my system those paths are wrong. I don't install the .deb, or the .rpm. Typically, I install software my distribution packages with commands like nix-env -iA nixos.ripgrep. These commands wind up pulling in runtime dependencies (or build dependencies for a fallback build) and place the “package” in /nix/store. This works very well for appropriately licensed and distributed software – but the closed source ones.. they take the cake for adventures to places I didn't contemplate going.

These drivers, admittedly, were not closed source. So? Was it easy? Not a walk in the park, but not too bad. I've dealt with worse on NixOS.

The Linux solution published by the fine folks at Blackmagic Design, included a few utilities, helper executables, and, of course, the kernel module for the PCIe card itself. There's a separate SDK vended too – it's needed for building ffmpeg with decklink support, but that's very out of scope.

Back to the kernel module.

This expression handles a couple things:

  • unpack the assumptive tarball (it's basically an rpm but not)
  • patching the upstream sources (to actually work.. thanks to Arch Linux maintainers here!)
  • building the kernel module (it's a more-or-less standard process)
  • ripping out “impure” [^3] or unneeded references (using nuke-refs)

I'm not explaining Nix expressions in this go, maybe another time. I'm continuing on through anyway.

{ stdenv
, fetchpatch
, nukeReferences
, linuxPackages
, kernel ? linuxPackages.kernel
, version
, src
}:

stdenv.mkDerivation {
  name = "blackmagic-${version}-module-${kernel.modDirVersion}";
  inherit version;

  buildInputs = [ nukeReferences ];

  kernel = kernel.dev;
  kernelVersion = kernel.modDirVersion;

  inherit src;
  patches = [
    (fetchpatch {
      name = "fix-get_user_pages-and-mmap_lock.patch";
      url = "https://aur.archlinux.org/cgit/aur.git/plain/02-fix-get_user_pages-and-mmap_lock.patch?h=decklink&id=8f19ef584c0603105415160d2ba4e8dfa47495ce";
      sha256 = "08m4qwrk0vg8rix59y591bjih95d2wp6bmm1p37nyfvhi2n9jw2m";
    })
    (fetchpatch {
      name = "fix-have_unlocked_ioctl.patch";
      url = "https://aur.archlinux.org/cgit/aur.git/plain/03-fix-have_unlocked_ioctl.patch?h=decklink&id=8f19ef584c0603105415160d2ba4e8dfa47495ce";
      sha256 = "0j9p62qa4mc6ir2v4fzrdapdrvi1dabrjrx1c295pwa3vmsi1x4f";
    })
  ];

  postUnpack = ''
    cd */usr/src
    sourceRoot="$(pwd -P)"
  '';

  buildPhase = ''
    cd $sourceRoot/blackmagic-''${version}*/
    # missing some "touch" commands, make sure they exist for build.
    touch .bmd-support.o.cmd
    make -C $kernel/lib/modules/$kernelVersion/build modules "M=$(pwd -P)"

    cd $sourceRoot/blackmagic-io-''${version}*/
    # missing some "touch" commands, make sure they exist for build.
    touch .blackmagic.o.cmd
    make -C $kernel/lib/modules/$kernelVersion/build modules "M=$(pwd -P)"

    cd $sourceRoot
  '';

  installPhase = ''
    mkdir -p $out/lib/modules/$kernelVersion/misc
    for x in $(find . -name '*.ko'); do
      nuke-refs $x
      cp $x $out/lib/modules/$kernelVersion/misc/
    done
  '';

  meta.platforms = [ "x86_64-linux" ];
}

Building the kernel module was a piece of cake, really. The meat of the “interesting parts” here was a snippet I kept from another round of kernel module hackery (oh, FireWire..):

    make -C $kernel/lib/modules/$kernelVersion/build modules "M=$(pwd -P)"

The above line builds the appropriate kernel module with, and for, the provided kernel. That's pretty typical of any given kernel module – in tree and out of tree modules. When the above derivation is realised (ie: built), the derivation (ie: a package, but not in the traditional sense) outputs a kernel module – the .ko – such that it'll appear at /run/current-system/kernel-modules/lib/modules/5.9.10/misc. modprobe is configured on NixOS hosts to use this directory, so all is well!

But, that's not all. I mentioned that there were helper executables. They're needed in order to setup a capture device when the PCI probe matches one by way of udev. Without the helpers, the device is halfway initialized – software that's capable of using the correct IOCTLs can't even see the devices until the helpers have a chat with the kernel module.

In full glory, here's the tools. Just the tools. There's still the SDK too..

{ stdenv
, autoPatchelfHook
, makeWrapper
, libGL, libGLU
, libuuid
, dbus_libs
, alsaLib
, xorg
, fontconfig
, freetype
, glib
, version
, src
, mediaexpress ? null
}:

stdenv.mkDerivation {
  pname = "blackmagic-tools";
  inherit version;

  inherit src mediaexpress;

  donStrip = true;

  nativeBuildInputs = [ makeWrapper autoPatchelfHook ];
  buildInputs = [ libGL libGLU libuuid dbus_libs freetype fontconfig glib alsaLib ]
                ++ (with xorg; [libxcb libXrender libICE libX11 libXinerama libXrandr libSM ]);


  postUnpack = ''
    if [[ -s "$mediaexpress" ]]; then
      tar -C $sourceRoot --strip-components=1 -xf $mediaexpress
    fi
  '';

  doBuild = false;

  installPhase = ''
    bins=( $(cd usr/bin; ls) )
    # Add the helpers to bin. They're needed by udev rules and the supporting
    # systemd service.
    bins+=( DesktopVideoNotifier DesktopVideoHelper )
    # Prune vended systemd units and the symlinked bin stubs.
    rm -rf usr/lib/systemd usr/bin

    cp -R usr $out

    # Replace symlinks with wrappers to include dynamicly dlopen'd libraries.
    # patchelf may corrupt the executables when adding a static entry that would
    # normally influence the RPATH.
    libBin=$out/lib/blackmagic/DesktopVideo
    for x in "''${bins[@]}"; do
      towrap="$out/lib/blackmagic/MediaExpress/$x"
      if ! [[ -x "$towrap" ]]; then
        towrap="$libBin/$x"
      fi
      makeWrapper $towrap $out/bin/$x \
        --prefix LD_LIBRARY_PATH ':' $out/lib \
        --prefix QT_PLUGIN_PATH ':' $libBin/plugins
    done

    # Need to substitute the executable path in these rules to use the $out/bin
    # path.
    mkdir -p $out/lib/udev/rules.d
    substitute \
      etc/udev/rules.d/55-blackmagic.rules \
      $out/lib/udev/rules.d/55-blackmagic.rules \
      --replace /usr/lib/blackmagic/DesktopVideo/DesktopVideoNotifier \
               $out/lib/blackmagic/DesktopVideo/DesktopVideoNotifier
  '';

  preFixup = ''
    # add $out/lib to the RPATH set on executables to use bundled version of
    # Qt5.
    runtimeDependencies+=" $out"
  '';
}

The tools provided the helpful little viewer and card setting utilities, right next to DesktopVideoNotifier and DesktopVideoHelper which provide the other half of capture card initialization. So, because I was being pragmatic, I lumped together these tools and did “brain surgery” in the context of executables by using patchelf (by way of autoPatchelf): the bundled shared libraries and expected runtime libraries, which were expected to be present like the popular crowds, were all in fact shoehorned into the executables' RPATH. And then wrapped.

As it turns out, wrappers are a good thing. Nixpkgs uses them liberally to improve the flexibility and reusability of its more build-time-consuming derivations (not as a goal per-se). This saves on build time and also disk space – if you wrap the thing you want to customize in a way that permits stubbing or subbing out dependencies, runtime configuration, or plain 'ol dependency resolution then you can easily change your mind later on. If you read through packages in the Nixpkgs repository, you'll come to see that its not uncommon to have packages several layers deep of derivations. Often times they're layered specifically to allow users to override some characteristic or improve composition of multiple derivations to produce a final useful environment.

You might have noticed there's missing.. everything.. that should have connected these up to be something. In this case, you'll have to deal with it. I returned the capture card after struggling to keep it and the devices plugged into it on the correct output/input formats – on top of the whole “we're NVIDIA and what is a GPL?” thing here in November of 2020 which made CUDA/NVENC workloads non-functional with Linux 5.9.

I've got the code hanging around and arguably should flesh this out more. I used more words than planned anyhow, so this'll be it.

[^1]: I'm not going to even try to quantify

[^2]: This is cool stuff, there's lot's to read on this topic, foo

[^3]: Just search for impure in the Nix manual and Nixpkg's, purity is a nuisance to the solution that Nix proposes.