Practicing Pragmatism

Today I needed to get some statistics for memory and storage analysis. There are a significant number of records that we'll be processing and an even larger set that represents every record ever observed!

The design is coming along well and we have a path to tackle this. However, I wanted to establish some expectation with regard to the memory required at runtime and the storage required over time.

Tentatively, the plan is to write out records not unlike the OCI Content descriptor, so these records are actually line delimited serialized JSON. Each entry is a file read in from a filesystem and records its path, size, and content digest.

That's the background.

Now, I needed to process a set of these records to determine what a typical record size is to scale up to a larger theoretical set size. I was going to hack it out with awk (because I have a tendency to do that) but found myself instead looking into GNU datamash.

It does everything I want to and gives my my quantiles as well!

datamash --headers count 1 min 1 max 1 median 1 perc:99 1 < records.sizes | column -t

You specify which stat and which field you want to display that stat for and voilà:

count(101)  min(101)  max(101)  median(101)  perc:99(101)
6329640     69        539       260          452

Now I have readily accessible and usable data to work with! Neat.

I can't say I know everything about code reviews, but I have learned a thing or two doing them at a pretty large company over the last half-decade with a variety of peers. The biggest takeaways, if you find nothing more, that I encourage you to apply to your code reviews:

  • ask questions
  • assume the best of others
  • be honest, be respectful

ask questions

Asking questions is a great way to acknowledge & explore the solution/change that an author is presenting in the code review. What's unfortunate is that I'm (anecdotally) seeing less conversation on code reviews in the past few years in favor of moving quickly and keeping the velocity up up up. Questions that could have been asked during code review – ones that might even ask “the obvious” – can eliminate frustrating late night conversations had after being paged! Avoiding a misstep isn't the only benefit to asking questions.

The biggest benefit to asking questions – almost any question during code review – is that not only do the participants get their questions answered, but it also disperses the knowledge being discussed. Think “ideal university lecturer with lots of engagement”: everyone wins (if they're paying attention).

Reflecting on this a bit right now, I think we're beginning to focus too heavily on the “do it now” part of software development to even consider encouraging questions that folks new to a team, community, or company might ask. I'm biased towards wanting to, myself, understand and to share what I know with others – I enjoy teaching and helping others grow – so I'm nearly always inclined to keep folks on the same page, even at the cost of velocity. Why? I like folks on the same page because then that investment pays off in dividends in being able to take on more and more complex projects. It builds a culture of understanding and mentorship between engineers. Most of all, investing time to establish a shared understanding means that any/all of the engineers can tackle challenges with skill, expertise, and curiosity.

assume the best of others

You know who (probably|most-likely|absolutely) isn't intentionally throwing away an error that they should be handling? That person on the other end of the code review.

Personally speaking, I've seen a few different faces on the receiving end of code reviews: pedantry, unhealthy/stubborn skepticism, blatant disregard for shared goals, ambivalence, and total engagement. They all have their drawbacks and not all of them can be constructively communicated with.

Can you imagine folks thinking the worst of you, your code, and decisions that led you there? If you've been in the industry and seen this: I'm sorry, that sucks for you and sucks for the folks involved (whether they know it or not, its their loss as well). When it happens in a review, you can feel it. To do anything other than assume the best is to allow communication to be suspect to negative biases, inter-personal challenges, or – more innocently – plain & simple disengagement.

I implore every reviewer to assume the best of the code review's author & their intentions. Point out flaws, but do not assume anything about those flaws without starting a conversation to clear up a misunderstanding or resolve a latent bug!

Existing hang ups are difficult or impossible to avoid in some cases, and even then both the author and reviewers can balance out if they're all (at a minimum) trying to be positive and assume the best of the author. An entire team working to keep that “good energy” going is bound to pour that back into themselves & new engineers. In the same way that companies like Amazon have found success modeling their strategy around a virtuous cycle, so can code reviews invest into their participants by leaving every person free (free of the burden of assumptions) to ask questions, to be wrong (and to be right!), and teach each other in the process.

be honest, be respectful

Honestly, I have no credentials to speak to the psychology of this all. Take this with a grain of salt.

In my experience, balancing both honesty and respect in your words is the hardest part to practice and use in the code review.

To be honest with yourself (in that you speak your mind, your thoughts, and voice your intuition as constructive feedback in the review) might mean writing Typing this function's error handling isn't great (which you might honestly think and judge the code as such) is still a ways from I think we need to handle a few more edge cases in this function. In this case .... Granted: yes, there are more words. That's not a bad thing, but it can be if you wind up saying nothing. The tradeoff you are making is to trade time for consciously written words – the return is great: you're not starting an argument about the code and, ideally, there's no mistaking your words as making a slight against the author.

Writing positive criticisms is a wordsmithing artform – it'll take practice and time to get the feel for. I recommend adopting the principals (and framework) of Nonviolent Communication – it's largely a superset of the above points in that its goals are specifically intended to foster empathy and explicitly not to avoid disagreements. The psychology behind the concept, to me anyway, seems to be well in line with code review! You need to be able to disagree, but also have constructive, productive, and collaborative conversation immediately follow. For those still not sold, here's an excerpt from the above linked Wikipedia article on Nonviolent Communication (NVC):

It is not a technique to end disagreements, but rather a method designed to increase empathy and improve the quality of life of those who utilize the method and the people around them.

I've personally seen conversation thrive in a setting of NVC, much more than one without (or with a bias towards terse corrections) – I can attest that not only does it improve (my, an engineer using NVC) quality of life but also the code I and my coworkers are working on. The technique is certainly not for everyone and every situation (or every code review for that matter) but it is a helpful tool to kick things off.

Regardless of what communication tools and patterns you reach for, I encourage you to remain honest – freely providing feedback and your thoughts – while respecting others with your delivery. Internalizing communication habits take time and will be a conscious struggle until its not – but the effort is worth it. For everyone.

summary

So, this is a bit of a rant, but I needed to get my own thoughts collected. In the last 2 years, I've switched teams, had engineers come and go on the team. In one month I even saw a NOTABLE improvement to the team's code review habits and feedback when a frequently argumentative engineer left the team.

Not to toot my own horn, but the folks on my current team have privately messaged me and thanking me for cracking the door on code reviews for them. I've had discussions and brief exchanges with folks that largely find the same: having positive interactions that leave room for questions, for discussion, that also do not belittle (whether by strongarming or words) is a damn sight better than one with no discussion and certainly better than one with a one sided or negative review.

I'm convinced that engineers teach other engineers how to talk. So here's me throwing my part in for that effort!

It seems like when you want to do something unusual that one finds another human's interesting configuration and workflows that they found worked for them in their time of need.

The best part is that their pain is shared and skipped by others. In my case, there was this StackOverflow Q/A (How to tell git which private key to use?) that shared the (at the time) new configuration item to set Git's SSH command that's used for pushes and pulls.

However, their command wasn't 100% there for me. I generally use a PGP hardware key with an authentication subkey when using SSH to log into my own hosts and also for VCS forges, like GitHub. This doesn't work very well, at all, for mechanical processes where automation doesn't have access to my hardware key or pin to unlock the hardware token.

Ideally, I'd have a scheduled job that would be able to authenticate and push a snapshot of config to GitHub periodically. Hence finding the aforementioned SO Q/A and wanting to apply this for myself. The wrench was that when I was sudo-ing as the user to test things out is that it “sees” my hardware token (because it uses an SSH Agent process to handle cryptographic operations) and gracefully falls back to using an on-disk machine-specific SSH key.

So, I got to take advantage of the SO's answerer's suggestion to use core.sshCommand by setting further options for my use case:

Where they suggested:

git config core.sshCommand "ssh $HOME/.ssh/id_rsa_example -F /dev/null" 

I needed to add an option to disable the use of any SSH agent process:

git config core.sshCommand "ssh -o IdentityAgent=none -i $HOME/.ssh/id_rsa_example -F /dev/null" 

Flags used here (see man page):

With that, I can run my SSH commands without having to see a prompt for my hardware token and/or failure to communicate with my user's private SSH agent socket.

Nice. And this allows simple per-repository tuning because its “just” git-config. Astute readers will also note that this means you can use further conditional configuration to set per-remote and per-ref configuration even!

These are pragmatic bits I found this morning. I hope you find a practical and pragmatic use for them as well. Cheers!

I was reading lwn articles recently and saw a post pointing to and quoting a well versed walk through of processes' lifecycle on Linux.

Their excerpt included a bit about setting LD_SHOW_AUXV to make ld (I think? I opened the tab in the background to read...). This was immediately interesting to me: any time that someone is sharing their own hard earned understanding LD.. I listen well! There's a lot of engineering behind the likes of ld and linux (naturally!) – in this case, the tip to use this environment variable was immediately at hand:

I ran sh with the variable exported on my system:

❯ LD_SHOW_AUXV=1 /bin/sh 
AT_SYSINFO_EHDR:      0x7ffed19db000
AT_HWCAP:             bfebfbff
AT_PAGESZ:            4096
AT_CLKTCK:            100
AT_PHDR:              0x400040
AT_PHENT:             56
AT_PHNUM:             11
AT_BASE:              0x7f5004b68000
AT_FLAGS:             0x0
AT_ENTRY:             0x41db30
AT_UID:               1000
AT_EUID:              1000
AT_GID:               100
AT_EGID:              100
AT_SECURE:            0
AT_RANDOM:            0x7ffed19c5f89
AT_HWCAP2:            0x2
AT_EXECFN:            /bin/sh
AT_PLATFORM:          x86_64

Neat! I know most of these short and terse identifiers with others that are new to me. These are the edges of the engineering world I like to find, avenues to explore further to integrate and apply to my own work.

Well, enough of that! What else? Let's try another command, after all the environment variable is exported. How about true – that's pretty trivial.

sh-4.4$ true

But also boring. There's no output. Well, that's because we're probably using bash's builtin true “function”.

sh-4.4$ /run/current-system/sw/bin/true
AT_SYSINFO_EHDR:      0x7ffd951e6000
AT_HWCAP:             bfebfbff
AT_PAGESZ:            4096
AT_CLKTCK:            100
AT_PHDR:              0x400040
AT_PHENT:             56
AT_PHNUM:             11
AT_BASE:              0x7eff8120f000
AT_FLAGS:             0x0
AT_ENTRY:             0x4089c0
AT_UID:               1000
AT_EUID:              1000
AT_GID:               100
AT_EGID:              100
AT_SECURE:            0
AT_RANDOM:            0x7ffd95067819
AT_HWCAP2:            0x2
AT_EXECFN:            /run/current-system/sw/bin/true
AT_PLATFORM:          x86_64

Note: my output says things like /run/current-system/sw/bin/true with that long path because I use NixOS and that's just where its located. Pretend lines like those say /bin/true or /usr/bin/true if that tickles your fancy.

I want to see the opposite now, where nothing is printed out with the environment still configured, because I suspect that this is implemented in glibc based on the little I've actually read of the article (I will read it, eventually).

To do this, I wrote a quick Go program to say hi that I can use static compilation to avoid linking to a libc:

package main

func main() { print("hi there\n") }

I built it:

CGO_ENABLED=0 go build static-go-bin.go

Checked it for riddles:

sh-4.4$ ldd ./static-go-bin
	not a dynamic executable

And then ran it with the environment variable set:

❯ LD_SHOW_AUXV=1 ./static-go-bin 
hi there

Hrm. Okay, nothing printed, but that's not conclusive. We still need to know if its ld or something that ld loaded that makes something print.

You know what, I'm going to stop here and just go read the article! That seems most pragmatic at this point.

I had a TON of configuration files that I needed to deal with at work. After writing myself a handy tool to produce the “input” configuration details, I figured I'd just use some quick and dirty Ruby (and ERB) to write the necessary files:

#!/usr/bin/env ruby
require 'json'
require 'erb'

template_file = "./template.toml.erb"
output_file_name = "./configuration/foo/conf-%{id_token}.toml"

# given a JSON array through STDIN of [ {conf}, {conf} ]
confs = JSON.parse($stdin.read, symbolize_names: true).map{|c| OpenStruct.new c }

# we can prepare the ERB template
template = ERB.new(File.read(template_file), 0, "%-"))

# finally, write out files for the conf objects
confs.each do |c|
  result = template.result(c.instance_eval("binding"))
  File.write(output_file_name % c, result)
end

tl;dr: try out C-x 5 5! (will be available in 28.1)

Emacs' frames are mighty powerful, though they can easily turn into a mess of windows.. but that's not what I wanted to share here.

Today, I wanted to open up an Info page in its own frame. That way I could keep it up and try out some command that typically mucks about with the current window setup. So, to avoid that a new frame would be handy – I thought – and so I went looking into the C-x 5 prefix's bindings:

C-x 5 C-h:

Global Bindings Starting With C-x 5:
key             binding
---             -------

C-x 5 C-f       find-file-other-frame
C-x 5 C-o       display-buffer-other-frame
C-x 5 .         xref-find-definitions-other-frame
C-x 5 0         delete-frame
C-x 5 1         delete-other-frames
C-x 5 2         make-frame-command
C-x 5 5         other-frame-prefix
C-x 5 b         switch-to-buffer-other-frame
C-x 5 d         dired-other-frame
C-x 5 f         find-file-other-frame
C-x 5 m         compose-mail-other-frame
C-x 5 o         other-frame
C-x 5 p         project-other-frame-command
C-x 5 r         find-file-read-only-other-frame

This one caught my eye – its new and coming in Emacs 28.1!

C-x 5 5         other-frame-prefix

So, to get my new frame with my desired Info page up, I hit the following keys:

C-x 5 5 M-x i n f o <RET>

That's it.


I keep up with the development branch of Emacs (mirrored on GitHub) on all my machines – and its coming features like this that remind me how much folks still care about Emacs and why I ought to keep up with the latest development efforts. Between general API improvements and the performance gains of Native Compilation, Emacs is a worthy tool to belong to every and any pragmatic fool (I dare you to be pragmatic when elisp is at your fingertips).

TIL about sysrc(8)

The sysrc tool is very handy for administering FreeBSD hosts. I've never shied away from vim /etc/rc.conf (though really emacs /etc/rc.conf for tidy editing, I'm an Emacser) so its not a tool that would have been sought out in my case. That said, any programmatic access to the system configuration is always welcomed with open arms.

Nevertheless, having more tools in the pragmatic belt makes for a better belt – as long as it makes things more practical and pragmatic in approach. In my case, I wanted to setup a tFTP server to allow me to install a fresh image on my network hardware. They require a tftp://$serverip/junos/$release_blob.tgz URL to load themselves up from and this means.. well that you need a tFTP server.

So, to set up the server we'll need have inetd start it and then start (or restart) inetd:

# Enable inetd services
sysrc inetd_enable=YES
# Configure tFTP to start (by inetd)
sed -i '' -E 's/#(tftp.*udp[[:blank:]])/\1/' /etc/inetd.conf
# Start it up!
service inetd start

My host uses ZFS as its root filesystem, so I also added a filesystem where /tftpboot lives:

zfs create -o mountpoint=/tftpboot $rootfs/tftpboot

With that, you can place files in /tftpboot that'll be retrievable by clients. Neat & simple.


TL;DR: An earlier version of the software was erroneously released (15.1R{3,5}) and later recalled.. I bet the same is true for the version I attempt to install here (15.1R7.9).


Yeah, I have some aging network equipment that's entirely overkill for my home network. The switch is an EX3200-48T which means it has plenty of ports and a little bit of power (over ethernet :)). I use this as my top-of-rack switch.. which is basically a mish-mash of WAN, LAN, and IoT traffic all switched and routed there (partially).

I found that it was giving some odd error messages while configuring the switch. I turned first to see what software versions I had: they were old. And the existing configuration? It was from me, but it was outdated & from another set up where the noise wasn't an issue.

Seeing as I was going to wipe and start clean slate with config, I figured I may as well also give it a fresh installed copy of JunOS. On blogs and, most importantly, Juniper's website the steps outlined included a “hit space when the loader tells you to” step. I was slightly confused by this as the loader immediately loads a “kernel” and continues on its way. So where the heck do you hit space? And where was this prompt at?

Usually, I'm pretty patient for these things, but this one got me pretty well:

The boot was loading some kernel, but that kernel is the loader that's referenced by docs. I've copied the below from a boot where I managed to hit space during the loader's execution which allowed me to run the install command (specifically install --format file:///$jinstall_pkg_tgz).

In other boots, this console was chock full of output and swiftly moved into starting the system proper. I suppose my eyes need another exam. Or that I should have used more time to examine with my eyes – probably that one.


The loader loading the loader where hitting space does something:

U-Boot 1.1.6 (Mar 28 2011 - 04:05:40)

Board: EX3200-48T 4.14
EPLD:  Version 10.0 (0x82)
DRAM:  Initializing (512 MB)
FLASH: 8 MB

Firmware Version: --- 01.00.00 ---
USB:   scanning bus for devices... 3 USB Device(s) found
       scanning bus for storage devices... 2 Storage Device(s) found

ELF file is 32 bit
Consoles: U-Boot console  

FreeBSD/PowerPC U-Boot bootstrap loader, Revision 2.4
(builder@dagmath.juniper.net, Mon Mar 28 01:49:54 UTC 2011)
Memory: 512MB
bootsequencing is enabled
bootsuccess is set
new boot device = disk0s2:
Loading /boot/defaults/loader.conf 
/kernel data=0xb71e7c+0xd12b0 syms=[0x4+0x9dce0+0x4+0xec41d]

# [there's a significant gap in time here]

Hit [Enter] to boot immediately, or space bar for command prompt.

Type '?' for a list of commands, 'help' for more detailed help.
loader>

Here's the outcome for those interested: install --format file:///$jinstall_pkg_tgz

loader> install --format file:///jinstall-ex-3200-15.1R7.9-domestic-signed.tgz
Package /jinstall-ex-3200-15.1R7.9-domestic-signed.tgz is signed...
/kernel data=0x69b300+0x93624 syms=[0x4+0x67960+0x4+0xa051b]
Kernel entry at 0x800000c0 ...
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1996-2018, Juniper Networks, Inc.
All rights reserved.
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
JUNOS 15.1R7.9 #0: 2018-09-11 05:30:26 UTC
    builder@watatsumi.juniper.net:/volume/build/junos/15.1/release/15.1R7.9/obj/powerpc/junos/bsd/kernels/INSTALL-EX/kernel
can't re-use a leaf (all_slot_serialid)!
Timecounter "decrementer" frequency 37500000 Hz quality 0
cpu0: Freescale e500v2 core revision 2.3
cpu0: HID0 80004080<EMCP,TBEN,EN_MAS7_UPDATE>
real memory  = 490733568 (468 MB)
avail memory = 479338496 (457 MB)
ETHERNET SOCKET BRIDGE initialising
Initializing EXSERIES properties ...
nexus0: <Powerpc Nexus device>
ocpbus0: <on-chip peripheral bus> on nexus0
openpic0: <OpenPIC in on-chip peripheral bus> iomem 0xfef40000-0xfef600b3 on ocpbus0
memctl0: <mpc85xx memory ECC monitor> iomem 0xfef20000-0xfef20e5b,0xfef02000-0xfef02e5b irq 32,34 on ocpbus0
ECC not enabled to report errors 0xc3000000
device_attach: memctl0 attach returned 6
i2c0: <MPC85XX OnChip i2c Controller> iomem 0xfef03000-0xfef03014 irq 59 on ocpbus0
i2c1: <MPC85XX OnChip i2c Controller> iomem 0xfef03100-0xfef03114 irq 59 on ocpbus0
lbc0: <Freescale Local Bus Controller> iomem 0xfef05000-0xfef05fff,0xff000000-0xffffffff irq 22 on ocpbus0
cfi0: <AMD/Fujitsu - 8MB> iomem 0xff800000-0xffffffff on lbc0
syspld0 iomem 0xff000000-0xff00ffff on lbc0
uart0: <16550 or compatible> iomem 0xfef04500-0xfef0450f irq 58 on ocpbus0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> iomem 0xfef04600-0xfef0460f irq 58 on ocpbus0
tsec0: <eTSEC ethernet controller> iomem 0xfef24000-0xfef24fff irq 45,46,50 on ocpbus0
tsec0: hardware MAC address 40:b4:f0:63:78:ff
miibus0: <MII bus> on tsec0
e1000phy0: <Marvell 88E1112 Gigabit PHY> on miibus0
e1000phy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto
pcib0: <Freescale MPC8544 PCI host controller> iomem 0xfef08000-0xfef08fff,0xf0000000-0xf3ffffff on ocpbus0
pci0: <PCI bus> on pcib0
pci0: <serial bus, USB> at device 18.0 (no driver attached)
ehci0: <Philips ISP156x USB 2.0 controller> mem 0xf0001000-0xf00010ff irq 22 at device 18.2 on pci0
usb0: EHCI version 1.0
usb0: <Philips ISP156x USB 2.0 controller> on ehci0
usb0: USB revision 2.0
uhub0: Philips EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
umass0: STMicroelectronics ST72682  High Speed Mode, rev 2.00/2.10, addr 2
umass1: CHIPSBNK v3.3.8.8, rev 2.00/1.00, addr 3
pcib1: <Freescale MPC8544 PCI Express host controller> iomem 0xfef0a000-0xfef0afff,0xe0000000-0xe3ffffff,0xec000000-0xec0fffff irq 42 on ocpbus0
pci1: <PCI bus> on pcib1
pcib2: <PCI-PCI bridge> at device 0.0 on pci1
pci2: <PCI bus> on pcib2
pci2: <memory> at device 0.0 (no driver attached)
pcib3: <Freescale MPC8544 PCI Express host controller> iomem 0xfef0b000-0xfef0bfff,0xe8000000-0xebffffff,0xec200000-0xec2fffff irq 43 on ocpbus0
pci3: <PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 0.0 on pci3
pci4: <PCI bus> on pcib4
pci4: <memory> at device 0.0 (no driver attached)
Initializing product: 38 ..
###PCB Group initialized for udppcbgroup
###PCB Group initialized for tcppcbgroup
md0: Preloaded image </isofs-install-ex> 22267904 bytes at 0x808367a8
da1 at umass-sim1 bus 1 target 0 lun 0
da1: <CHIPSBNK v3.3.8.8 5.00> Removable Direct Access SCSI-2 device 
da1: 40.000MB/s transfers
da1: 3920MB (8028160 512 byte sectors: 255H 63S/T 499C)
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <ST ST72682 2.10> Removable Direct Access SCSI-2 device 
da0: 40.000MB/s transfers
da0: 1000MB (2048000 512 byte sectors: 64H 32S/T 1000C)
Kernel thread "wkupdaemon" (pid 43) exited prematurely.
Trying to mount root from cd9660:/dev/md0
[: -eq: unexpected operator
1+0 records in
1+0 records out
512 bytes transferred in 0.000285 secs (1795555 bytes/sec)
Media check on da0 on ex platforms
Formatting installation disk...
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 269.710252 secs (3887787 bytes/sec)
Computing slice and partition sizes for /dev/da0 ...
Formatting target media /dev/da0 ...
Preparing to create slices on /dev/da0
/dev/da0: 2048000 sectors [C:1000 H:64 S:32 SS:512]
Shrinking slice 1 by 256 blocks for alignment
1+0 records in
1+0 records out
512 bytes transferred in 0.000290 secs (1766023 bytes/sec)
Creating slices:
g c1000 h64 s32
p 1    0xA5 256 382720
p 2    0xA5 382976 382976
p 3    0xA5 765952 1024000
p 4    0xA5 1789952 258048
a 1
******* Working on device /dev/da0 *******
fdisk: invalid fdisk partition table found
Computing layout of partitions in /dev/da0s1...
Shrinking partition a by 1792 blocks for alignment
Labeling /dev/da0s1:
bsdlabel: write to disk label supressed - label was as follows:
# /dev/da0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:   380672      256    unused        0     0       
  c:   382720        0    unused        0     0         # "raw" part, don't edit
/dev/da0s1a: 185.9MB (380668 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 46.47MB, 2974 blks, 6016 inodes.
        with soft updates
super-block backups (for fsck -b #) at:
 32, 95200, 190368, 285536
Computing layout of partitions in /dev/da0s2...
Labeling /dev/da0s2:
bsdlabel: write to disk label supressed - label was as follows:
# /dev/da0s2:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:   382720      256    unused        0     0       
  c:   382976        0    unused        0     0         # "raw" part, don't edit
/dev/da0s2a: 186.9MB (382716 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 46.72MB, 2990 blks, 6016 inodes.
        with soft updates
super-block backups (for fsck -b #) at:
 32, 95712, 191392, 287072
Computing layout of partitions in /dev/da0s3...
Shrinking partition d by 256 blocks for alignment
Labeling /dev/da0s3:
bsdlabel: write to disk label supressed - label was as follows:
# /dev/da0s3:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  c:  1024000        0    unused        0     0         # "raw" part, don't edit
  d:   767744      256    unused        0     0       
  e:   256000   768000    unused        0     0       
/dev/da0s3d: 374.9MB (767740 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 93.72MB, 5998 blks, 12032 inodes.
        with soft updates
super-block backups (for fsck -b #) at:
 32, 191968, 383904, 575840
/dev/da0s3e: 125.0MB (255996 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 31.25MB, 2000 blks, 4096 inodes.
        with soft updates
super-block backups (for fsck -b #) at:
 32, 64032, 128032, 192032
Computing layout of partitions in /dev/da0s4...
Shrinking partition d by 256 blocks for alignment
Labeling /dev/da0s4:
bsdlabel: write to disk label supressed - label was as follows:
# /dev/da0s4:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  c:   258048        0    unused        0     0         # "raw" part, don't edit
  d:   128768      256    unused        0     0       
/dev/da0s4d: 62.9MB (128764 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 15.72MB, 1006 blks, 2048 inodes.
        with soft updates
super-block backups (for fsck -b #) at:
 32, 32224, 64416, 96608
[: -eq: unexpected operator
Installing disk1:/jinstall-ex-3200-15.1R7.9-domestic-signed.tgz
/dev/da0s3d: 374.9MB (767740 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 93.72MB, 5998 blks, 12032 inodes.
        with soft updates
super-block backups (for fsck -b #) at:
 32, 191968, 383904, 575840
Verified jinstall-ex-3200-15.1R7.9-domestic.tgz signed by PackageProductionRSA_2018 method RSA2048+SHA1
date: connect: Can't assign requested address
Checking package integrity...
Running requirements check first for jbundle-ex-3200-15.1R7.9-...
Running pre-install for jbundle-ex-3200-15.1R7.9-...
Installing jbundle-ex-3200-15.1R7.9- in /tmp/installer.tmp/pa4731.35/jbundle-ex-3200-15.1R7.9-domestic.x4731...
Running post-install for jbundle-ex-3200-15.1R7.9-...
Verified SHA1 checksum of fips-mode-powerpc-15.1R7.9.tgz
Verified SHA1 checksum of jboot-ex-15.1R7.9.tgz
Verified SHA1 checksum of jdocs-ex-15.1R7.9.tgz
Verified SHA1 checksum of junos-ex-15.1R7.9.tgz
Verified SHA1 checksum of junos-ex-3200-15.1R7.9.tgz
Verified SHA1 checksum of jweb-ex-15.1R7.9.tgz
Adding fips-mode-powerpc-15.1R7.9.tgz...
Adding jdocs-ex-15.1R7.9.tgz...
Adding junos-ex-3200-15.1R7.9.tgz..

And, guess what, my problem persists.

/usr/libexec/ld-elf.so.1: Shared object "libcmd-parser.so.1" not found, required by "jdhcpd"

Wanna know why?

This technical bulletin explains why: https://kb.juniper.net/InfoCenter/index?id=TSB17033&page=content

TL;DR: An earlier version of the software was erroneously released and later recalled.. I bet the same is true for the version I attempted to install (15.1R7.9).

Sigh.

Fun fact, having CDPATH exported will cause it to also emit any fuzzy matched to stdout.

I spent entirely too long digging into a Makefile where it had something clever like:

check-gofmt:
    $(eval HAS_DELTA := $(shell cd src && gofmt -l $(PKGS))
    if [[ -n "$HAS_DELTA" ]]; then echo "reformat your code please:\n$HAS_DELTA"; exit 1; fi 

Turns out that the path src didn't exist, but because I had mistakenly exported a CDPATH variable in my .bashrc (which has since been corrected):

export CDPATH="some/path:another/path/with/a/src/sub-directory"

There was a matching sub directory path that Make's $(shell cd src && ...) matched and emitted which made that clever if [[ -n "$HAS_DELTA" ]] really upset. When it failed, the message printed was the fuzzy-matched path itself.. which wasn't helpful in trying to diagnose things.

How did I diagnose this? Tracing!

# Turn on shell tracing to see where the output is printed:
SHELL = /usr/bin/bash -x

check-gofmt:
    # ...

Failed to load breakpoints for the current workspace with error: (error "Hash table data is not a list of even length") [209 times]

Emacs debugger frames

Debugger entered--Lisp error: (error "Hash table data is not a list of even length")
  read-from-string("#s(hash-table size 65 test equal rehash-size 1.5 r...")
  dap--read-from-file("/home/jake/.local/doom/etc/.dap-b...")
  dap--after-initialize()
  dap-mode(1)
  dap-auto-configure-mode(1)
  lsp-configure-buffer()
  #f(compiled-function (buf) #<bytecode 0x232e511>)(#<buffer options.go>)
  mapc(#f(compiled-function (buf) #<bytecode 0x232e511>) (#<buffer options.go> #<buffer options_test.go>))
  lsp--on-request(#s(lsp--workspace :ewoc nil :server-capabilities #<hash-table equal 21/21 0xabb17bd> :registered-server-capabilities (#s(lsp--registered-capability :id "workspace/didChangeWatchedFiles-630" :method "workspace/didChangeWatchedFiles" :options #<hash-table equal 1/1 0x1bc7c05>) #s(lsp--registered-capability :id "workspace/didChangeWatchedFiles-629" :method "workspace/didChangeWatchedFiles" :options #<hash-table equal 1/1 0xa76b4dd>)) :root "/home/jake/w/ft/src/FT" :client #s(lsp--client :language-id "go" :add-on? nil :new-connection (:connect #f(compiled-function (filter sentinel name environment-fn) #<bytecode 0x11cb031>) :test\? #f(compiled-function () #<bytecode 0x11caecd>)) :ignore-regexps nil :ignore-messages nil :notification-handlers #<hash-table equal 0/65 0x3a85529> :request-handlers #<hash-table equal 0/65 0x3a854a1> :response-handlers #<hash-table eql 0/65 0x3a853c1> :prefix-function nil :uri-handlers #<hash-table equal 0/65 0x3a85339> :action-handlers #<hash-table equal 0/65 0x3a852b1> :major-modes (go-mode go-dot-mod-mode) :activation-fn nil :priority 0 :server-id gopls :multi-root nil :initialization-options nil :custom-capabilities nil :library-folders-fn lsp-go--library-default-directories :before-file-open-fn nil :initialized-fn nil :remote? nil :completion-in-comments? t :path->uri-fn nil :uri->path-fn nil :environment-fn nil :after-open-fn #f(compiled-function () #<bytecode 0x11c3aa1>) :async-request-handlers #<hash-table equal 0/65 0x3a85229> :download-server-fn nil :download-in-progress? nil :buffers nil) :host-root nil :proc #<process gopls> :cmd-proc #<process gopls> :buffers (#<buffer options.go> #<buffer options_test.go>) :semantic-tokens-faces nil :semantic-tokens-modifier-faces nil :extra-client-capabilities nil :status initialized :metadata #<hash-table equal 0/65 0x4a26ef9> :watches #<hash-table equal 0/65 0x4a27021> :workspace-folders nil :last-id 0 :status-string nil :shutdown-action nil :diagnostics #<hash-table equal 4/65 0x4a27425> :work-done-tokens #<hash-table equal 0/65 0x4a277c5>) #<hash-table equal 4/4 0x266334d>)
  lsp--parser-on-message(#<hash-table equal 4/4 0x266334d> #s(lsp--workspace :ewoc nil :server-capabilities #<hash-table equal 21/21 0xabb17bd> :registered-server-capabilities (#s(lsp--registered-capability :id "workspace/didChangeWatchedFiles-630" :method "workspace/didChangeWatchedFiles" :options #<hash-table equal 1/1 0x1bc7c05>) #s(lsp--registered-capability :id "workspace/didChangeWatchedFiles-629" :method "workspace/didChangeWatchedFiles" :options #<hash-table equal 1/1 0xa76b4dd>)) :root "/home/jake/w/ft/src/FT" :client #s(lsp--client :language-id "go" :add-on? nil :new-connection (:connect #f(compiled-function (filter sentinel name environment-fn) #<bytecode 0x11cb031>) :test\? #f(compiled-function () #<bytecode 0x11caecd>)) :ignore-regexps nil :ignore-messages nil :notification-handlers #<hash-table equal 0/65 0x3a85529> :request-handlers #<hash-table equal 0/65 0x3a854a1> :response-handlers #<hash-table eql 0/65 0x3a853c1> :prefix-function nil :uri-handlers #<hash-table equal 0/65 0x3a85339> :action-handlers #<hash-table equal 0/65 0x3a852b1> :major-modes (go-mode go-dot-mod-mode) :activation-fn nil :priority 0 :server-id gopls :multi-root nil :initialization-options nil :custom-capabilities nil :library-folders-fn lsp-go--library-default-directories :before-file-open-fn nil :initialized-fn nil :remote? nil :completion-in-comments? t :path->uri-fn nil :uri->path-fn nil :environment-fn nil :after-open-fn #f(compiled-function () #<bytecode 0x11c3aa1>) :async-request-handlers #<hash-table equal 0/65 0x3a85229> :download-server-fn nil :download-in-progress? nil :buffers nil) :host-root nil :proc #<process gopls> :cmd-proc #<process gopls> :buffers (#<buffer options.go> #<buffer options_test.go>) :semantic-tokens-faces nil :semantic-tokens-modifier-faces nil :extra-client-capabilities nil :status initialized :metadata #<hash-table equal 0/65 0x4a26ef9> :watches #<hash-table equal 0/65 0x4a27021> :workspace-folders nil :last-id 0 :status-string nil :shutdown-action nil :diagnostics #<hash-table equal 4/65 0x4a27425> :work-done-tokens #<hash-table equal 0/65 0x4a277c5>))
  #f(compiled-function (proc input) #<bytecode 0x4a27041>)(#<process gopls> "Content-Length: 160641\15\n\15\n{\"jsonrpc\":\"2.0\",\"method...")

That's a lot of times. The upstream issue seems to suggest that this is caused by an signal interruption where the file became corrupt.

Okay, but I don't even use dap-mode even though the package is installed and configured in doom-emacs. My guess is that at some point I hit C-z (or something else) and Emacs didn't get to finish writing to this file (regardless of whether or not there was user data in there). When it loads the file – to then read it into a hash table – the data is found to be invalid or unusable. I'm not sure which.

However, the thing errored 209 times in a very tight loop.

Needless to say, Emacs wasn't happy. I made it happy by deleting the file that it was trying to load:

→ rm ~/.local/doom/etc/.dap-breakpoints