Practicing Pragmatism

Editor's note: I, the author and editor, was mistaken about the granularity at which the SNMP service is configured in an Unleashed set of APs – a summary is accessible at the primary controller AP IP via SNMP. See posted SNMP guide for your version (eg: https://support.ruckuswireless.com/documents/3435-ruckus-unleashed-200-9-ga-snmp-reference-guide/download)

The following is here for posterity – it does fiddle the right bits, but not in Unleashed

Found this while trying to get snmp enabled on all my APs rather than just the active controller AP at a given time (the web UI seems to configure the local to it snmp service).

Thanks to:

https://community.ruckuswireless.com/t5/Access-Points-Indoor-and-Outdoor/is-it-possible-to-enable-SNMP-on-Wireless-AP-and-get-it-via-the/td-p/44004

rkscli: get snmp
SNMP   enable             : disable
...
OK

Set up your SNMP details (check help set snmp) and then run the following to enable the service:

rkscli: set remote-mgmt snmp
OK

Now you're cooking with snmp.

rkscli: get snmp
SNMP   enable             : enable

There's been times where checking all elements of $PATH for something is necessary – perhaps for debugging, overriding purposes, or otherwise.

One liners are handy for this – I'm certainly not one to shy away from a neato shell one-liner to accomplish the job (or use find with some shell replacements). However! Code that you don't have to write is great in flow state (even if its a fun thought exercise that scratches an itch..) – ZSH lends itself to the task with a built in: whence (its the same builtin behind which). Check the manpage for the details, my copy (edited to fit nicely) says:

whence [ -vcwfpamsS ] [ -x num ] name ...

  For each name, indicate how it would be interpreted if used as a command
  name.

  If name is not an alias, built-in command, external command, shell
  function, hashed command, or a reserved word, the exit status shall be
  non-zero, and -- if -v, -c, or -w was passed -- a message will be written
  to standard output. (This is different from other shells that write that
  message to standard error.)

  whence is most useful when name is only the last path component of a
  command, i.e. does not include a `/'; in particular, pattern matching
  only succeeds if just the non-directory component of the command is
  passed.

  -v     Produce a more verbose report.

  -c     Print the results in a csh-like format. This takes precedence over -v.

  -w     For each name, print `name: word' where word is one of alias,
         builtin, command, function, hashed, reserved or none, according as
         name corresponds to an alias, a built-in command, an external
         command, a shell function, a command defined with the hash
         builtin, a reserved word, or is not recognised. This takes
         precedence over -v and -c.

  -f     Causes the contents of a shell function to be displayed, which
         would otherwise not happen unless the -c flag were used.

  -p     Do a path search for name even if it is an alias, reserved word,
         shell function or builtin.

  -a     Do a search for all occurrences of name throughout the command
         path. Normally only the first occurrence is printed.

  -m     The arguments are taken as patterns (pattern characters should be
         quoted), and the information is displayed for each command
         matching one of these patterns.

  -s     If a pathname contains symlinks, print the symlink-free pathname
         as well.

  -S     As -s, but if the pathname had to be resolved by following
         multiple symlinks, the intermediate steps are printed, too. The
         symlink re‐ solved at each step might be anywhere in the path.

  -x num Expand tabs when outputting shell functions using the -c option.
         This has the same effect as the -x option to the functions builtin.

So the typical usage I'm looking for is:

# show builtin & the command
whence -a time
# giving this on NixOS (first being a builtin)
time
/run/current-system/sw/bin/time

# or checking which `git` exectuables exist in (current, semi-contrived) PATH
whence -ap git
# giving this on Nix-on-NonNixOS host
/home/jake/.nix-profile/bin/git                                                                                                                                                                                  
/nix/store/50lch2g9xn0sw32b2r508d3hr6mfq07f-git-with-svn-2.41.0/bin/git                                                                                                                                            
/usr/bin/git

Nice!

I have seen Nix break itself (its DB) a few times. Once I even managed to wipe 90% of my /nix/store after not noticing the error and running nix-collect-garbage -d (to clear space).

Nix uses a database to track added paths and facilitate untrusted user's added paths with respect to the derivations used to create them. If this database is corrupted, then Nix has no idea about the paths that already exist on disk. In the case where I wiped a bunch of paths off my disk (deleting basically the entire system, the equivalent to rm -rf /usr on many other Linux distributions) the paths aren't at all registered, tracked, or considered valid.

So, Nix says “dunno what these are” and carries on. If you garbage collect during this time, well, those paths aren't registered and suspect to deletion.

You probably came for the repair. On to that bit now.

When needing to repair the database you'll need to either do one of two things:

  1. mangle whatever you have left of the database
  2. call it a loss and create a new database

I've done 1. It isn't fun. You'll need to check that the SQLite3 structures are valid and that constraints are still adhered to – this amounts to first running sqlite3 /nix/var/nix/db/db.sqlite "pragma integrity_check" to verify, and then dumping the database to individual insertions to remove the culprit (if you can find them all, which you can do, but holy heck is that tedious).

# dump whatever the database contains, to edit and create a new db with
sqlite3 -readonly /nix/var/nix/db/db.sqlite .dump > /nix/var/nix/db/dump-$EPOCHSECONDS.sql

# after editing, load it into a new database.. and cross your fingers
mv /nix/var/nix/db/{db.sqlite,db.sqlite.$EPOCHSECONDS}
sqlite3 /nix/var/nix/db/db.sqlite < /nix/var/nix/db/edited-dump.sql

Note: this is not guaranteed to work. Invalid paths and database constraints may plague the process over many iterations. You've been warned :)

After having done 1 more than once (yes, more than once), I've found its not worthwhile for my workstation purposes. In practice, the actual content is still on disk and won't be deleted if you're being cautious. Route 2 keeps these contents around – during which time I'd even bet the store may even still serve its purposes for you – and you instead import the paths into a fresh database. This does mean you'll lose some amount of information. I suspect that this means you'll lose the association between a store path, the producing derivation, and possibly compressed logs. I haven't confirmed this myself because I'm content gaining operability rather than worry about lost logs for successfully built derivations.

How do you go about taking route 2?

I'll show you, but first there's some amount of care to exercise:

# stash the bad db away. Just in case.
mkdir -p /nix/var/nix/db/prev.$$
mv /nix/var/nix/db/{schema,reserved,db.sqlite} /nix/var/nix/db/prev.$$

If you don't have nix on PATH still, then you might find it in /nix/store (or get a new copy from https://nixos.org/nix):

ls -1 /nix/store/*nix*/bin/nix

Pick one. If you can recall which version you were using – use that! – if not, then use one of them (preferably not a pre-release version – which might be why you landed here in the first place).

/nix/store/n8x6ig1yf8ffpa07mwvxg6b7ilrrvfy1-nix-2.4/bin/nix-store --init

And then, again, there's tedium or there's pragmatic ignorance (reasonable laziness perhaps?), either to live with unregistered paths (that are registered on demand as you use Nix) or to add the paths back to the database.

Again, I've done both. On a big builder host I'd take the time to import the paths, but on my workstation I'd rather just move on with my life. It isn't worth the time in that case because you will grow the database back to the learn about what you still have on disk. If the derivation continues to produce the same hash then Nix can and will import paths. Not in all cases, mind you, but still. It's a cost that some regions of the world with limited internet can't accept so I won't say its the right choice for everyone.

Today I needed to get some statistics for memory and storage analysis. There are a significant number of records that we'll be processing and an even larger set that represents every record ever observed!

The design is coming along well and we have a path to tackle this. However, I wanted to establish some expectation with regard to the memory required at runtime and the storage required over time.

Tentatively, the plan is to write out records not unlike the OCI Content descriptor, so these records are actually line delimited serialized JSON. Each entry is a file read in from a filesystem and records its path, size, and content digest.

That's the background.

Now, I needed to process a set of these records to determine what a typical record size is to scale up to a larger theoretical set size. I was going to hack it out with awk (because I have a tendency to do that) but found myself instead looking into GNU datamash.

It does everything I want to and gives my my quantiles as well!

datamash --headers count 1 min 1 max 1 median 1 perc:99 1 < records.sizes | column -t

You specify which stat and which field you want to display that stat for and voilà:

count(101)  min(101)  max(101)  median(101)  perc:99(101)
6329640     69        539       260          452

Now I have readily accessible and usable data to work with! Neat.

I can't say I know everything about code reviews, but I have learned a thing or two doing them at a pretty large company over the last half-decade with a variety of peers. The biggest takeaways, if you find nothing more, that I encourage you to apply to your code reviews:

  • ask questions
  • assume the best of others
  • be honest, be respectful

ask questions

Asking questions is a great way to acknowledge & explore the solution/change that an author is presenting in the code review. What's unfortunate is that I'm (anecdotally) seeing less conversation on code reviews in the past few years in favor of moving quickly and keeping the velocity up up up. Questions that could have been asked during code review – ones that might even ask “the obvious” – can eliminate frustrating late night conversations had after being paged! Avoiding a misstep isn't the only benefit to asking questions.

The biggest benefit to asking questions – almost any question during code review – is that not only do the participants get their questions answered, but it also disperses the knowledge being discussed. Think “ideal university lecturer with lots of engagement”: everyone wins (if they're paying attention).

Reflecting on this a bit right now, I think we're beginning to focus too heavily on the “do it now” part of software development to even consider encouraging questions that folks new to a team, community, or company might ask. I'm biased towards wanting to, myself, understand and to share what I know with others – I enjoy teaching and helping others grow – so I'm nearly always inclined to keep folks on the same page, even at the cost of velocity. Why? I like folks on the same page because then that investment pays off in dividends in being able to take on more and more complex projects. It builds a culture of understanding and mentorship between engineers. Most of all, investing time to establish a shared understanding means that any/all of the engineers can tackle challenges with skill, expertise, and curiosity.

assume the best of others

You know who (probably|most-likely|absolutely) isn't intentionally throwing away an error that they should be handling? That person on the other end of the code review.

Personally speaking, I've seen a few different faces on the receiving end of code reviews: pedantry, unhealthy/stubborn skepticism, blatant disregard for shared goals, ambivalence, and total engagement. They all have their drawbacks and not all of them can be constructively communicated with.

Can you imagine folks thinking the worst of you, your code, and decisions that led you there? If you've been in the industry and seen this: I'm sorry, that sucks for you and sucks for the folks involved (whether they know it or not, its their loss as well). When it happens in a review, you can feel it. To do anything other than assume the best is to allow communication to be suspect to negative biases, inter-personal challenges, or – more innocently – plain & simple disengagement.

I implore every reviewer to assume the best of the code review's author & their intentions. Point out flaws, but do not assume anything about those flaws without starting a conversation to clear up a misunderstanding or resolve a latent bug!

Existing hang ups are difficult or impossible to avoid in some cases, and even then both the author and reviewers can balance out if they're all (at a minimum) trying to be positive and assume the best of the author. An entire team working to keep that “good energy” going is bound to pour that back into themselves & new engineers. In the same way that companies like Amazon have found success modeling their strategy around a virtuous cycle, so can code reviews invest into their participants by leaving every person free (free of the burden of assumptions) to ask questions, to be wrong (and to be right!), and teach each other in the process.

be honest, be respectful

Honestly, I have no credentials to speak to the psychology of this all. Take this with a grain of salt.

In my experience, balancing both honesty and respect in your words is the hardest part to practice and use in the code review.

To be honest with yourself (in that you speak your mind, your thoughts, and voice your intuition as constructive feedback in the review) might mean writing Typing this function's error handling isn't great (which you might honestly think and judge the code as such) is still a ways from I think we need to handle a few more edge cases in this function. In this case .... Granted: yes, there are more words. That's not a bad thing, but it can be if you wind up saying nothing. The tradeoff you are making is to trade time for consciously written words – the return is great: you're not starting an argument about the code and, ideally, there's no mistaking your words as making a slight against the author.

Writing positive criticisms is a wordsmithing artform – it'll take practice and time to get the feel for. I recommend adopting the principals (and framework) of Nonviolent Communication – it's largely a superset of the above points in that its goals are specifically intended to foster empathy and explicitly not to avoid disagreements. The psychology behind the concept, to me anyway, seems to be well in line with code review! You need to be able to disagree, but also have constructive, productive, and collaborative conversation immediately follow. For those still not sold, here's an excerpt from the above linked Wikipedia article on Nonviolent Communication (NVC):

It is not a technique to end disagreements, but rather a method designed to increase empathy and improve the quality of life of those who utilize the method and the people around them.

I've personally seen conversation thrive in a setting of NVC, much more than one without (or with a bias towards terse corrections) – I can attest that not only does it improve (my, an engineer using NVC) quality of life but also the code I and my coworkers are working on. The technique is certainly not for everyone and every situation (or every code review for that matter) but it is a helpful tool to kick things off.

Regardless of what communication tools and patterns you reach for, I encourage you to remain honest – freely providing feedback and your thoughts – while respecting others with your delivery. Internalizing communication habits take time and will be a conscious struggle until its not – but the effort is worth it. For everyone.

summary

So, this is a bit of a rant, but I needed to get my own thoughts collected. In the last 2 years, I've switched teams, had engineers come and go on the team. In one month I even saw a NOTABLE improvement to the team's code review habits and feedback when a frequently argumentative engineer left the team.

Not to toot my own horn, but the folks on my current team have privately messaged me and thanking me for cracking the door on code reviews for them. I've had discussions and brief exchanges with folks that largely find the same: having positive interactions that leave room for questions, for discussion, that also do not belittle (whether by strongarming or words) is a damn sight better than one with no discussion and certainly better than one with a one sided or negative review.

I'm convinced that engineers teach other engineers how to talk. So here's me throwing my part in for that effort!

It seems like when you want to do something unusual that one finds another human's interesting configuration and workflows that they found worked for them in their time of need.

The best part is that their pain is shared and skipped by others. In my case, there was this StackOverflow Q/A (How to tell git which private key to use?) that shared the (at the time) new configuration item to set Git's SSH command that's used for pushes and pulls.

However, their command wasn't 100% there for me. I generally use a PGP hardware key with an authentication subkey when using SSH to log into my own hosts and also for VCS forges, like GitHub. This doesn't work very well, at all, for mechanical processes where automation doesn't have access to my hardware key or pin to unlock the hardware token.

Ideally, I'd have a scheduled job that would be able to authenticate and push a snapshot of config to GitHub periodically. Hence finding the aforementioned SO Q/A and wanting to apply this for myself. The wrench was that when I was sudo-ing as the user to test things out is that it “sees” my hardware token (because it uses an SSH Agent process to handle cryptographic operations) and gracefully falls back to using an on-disk machine-specific SSH key.

So, I got to take advantage of the SO's answerer's suggestion to use core.sshCommand by setting further options for my use case:

Where they suggested:

git config core.sshCommand "ssh $HOME/.ssh/id_rsa_example -F /dev/null"

I needed to add an option to disable the use of any SSH agent process:

git config core.sshCommand "ssh -o IdentityAgent=none -i $HOME/.ssh/id_rsa_example -F /dev/null"

Flags used here (see man page):

With that, I can run my SSH commands without having to see a prompt for my hardware token and/or failure to communicate with my user's private SSH agent socket.

Nice. And this allows simple per-repository tuning because its “just” git-config. Astute readers will also note that this means you can use further conditional configuration to set per-remote and per-ref configuration even!

These are pragmatic bits I found this morning. I hope you find a practical and pragmatic use for them as well. Cheers!

I was reading lwn articles recently and saw a post pointing to and quoting a well versed walk through of processes' lifecycle on Linux.

Their excerpt included a bit about setting LD_SHOW_AUXV to make ld (I think? I opened the tab in the background to read...). This was immediately interesting to me: any time that someone is sharing their own hard earned understanding LD.. I listen well! There's a lot of engineering behind the likes of ld and linux (naturally!) – in this case, the tip to use this environment variable was immediately at hand:

I ran sh with the variable exported on my system:

❯ LD_SHOW_AUXV=1 /bin/sh 
AT_SYSINFO_EHDR:      0x7ffed19db000
AT_HWCAP:             bfebfbff
AT_PAGESZ:            4096
AT_CLKTCK:            100
AT_PHDR:              0x400040
AT_PHENT:             56
AT_PHNUM:             11
AT_BASE:              0x7f5004b68000
AT_FLAGS:             0x0
AT_ENTRY:             0x41db30
AT_UID:               1000
AT_EUID:              1000
AT_GID:               100
AT_EGID:              100
AT_SECURE:            0
AT_RANDOM:            0x7ffed19c5f89
AT_HWCAP2:            0x2
AT_EXECFN:            /bin/sh
AT_PLATFORM:          x86_64

Neat! I know most of these short and terse identifiers with others that are new to me. These are the edges of the engineering world I like to find, avenues to explore further to integrate and apply to my own work.

Well, enough of that! What else? Let's try another command, after all the environment variable is exported. How about true – that's pretty trivial.

sh-4.4$ true

But also boring. There's no output. Well, that's because we're probably using bash's builtin true “function”.

sh-4.4$ /run/current-system/sw/bin/true
AT_SYSINFO_EHDR:      0x7ffd951e6000
AT_HWCAP:             bfebfbff
AT_PAGESZ:            4096
AT_CLKTCK:            100
AT_PHDR:              0x400040
AT_PHENT:             56
AT_PHNUM:             11
AT_BASE:              0x7eff8120f000
AT_FLAGS:             0x0
AT_ENTRY:             0x4089c0
AT_UID:               1000
AT_EUID:              1000
AT_GID:               100
AT_EGID:              100
AT_SECURE:            0
AT_RANDOM:            0x7ffd95067819
AT_HWCAP2:            0x2
AT_EXECFN:            /run/current-system/sw/bin/true
AT_PLATFORM:          x86_64

Note: my output says things like /run/current-system/sw/bin/true with that long path because I use NixOS and that's just where its located. Pretend lines like those say /bin/true or /usr/bin/true if that tickles your fancy.

I want to see the opposite now, where nothing is printed out with the environment still configured, because I suspect that this is implemented in glibc based on the little I've actually read of the article (I will read it, eventually).

To do this, I wrote a quick Go program to say hi that I can use static compilation to avoid linking to a libc:

package main

func main() { print("hi there\n") }

I built it:

CGO_ENABLED=0 go build static-go-bin.go

Checked it for riddles:

sh-4.4$ ldd ./static-go-bin
	not a dynamic executable

And then ran it with the environment variable set:

❯ LD_SHOW_AUXV=1 ./static-go-bin 
hi there

Hrm. Okay, nothing printed, but that's not conclusive. We still need to know if its ld or something that ld loaded that makes something print.

You know what, I'm going to stop here and just go read the article! That seems most pragmatic at this point.

I had a TON of configuration files that I needed to deal with at work. After writing myself a handy tool to produce the “input” configuration details, I figured I'd just use some quick and dirty Ruby (and ERB) to write the necessary files:

#!/usr/bin/env ruby
require 'json'
require 'erb'

template_file = "./template.toml.erb"
output_file_name = "./configuration/foo/conf-%{id_token}.toml"

# given a JSON array through STDIN of [ {conf}, {conf} ]
confs = JSON.parse($stdin.read, symbolize_names: true).map{|c| OpenStruct.new c }

# we can prepare the ERB template
template = ERB.new(File.read(template_file), 0, "%-"))

# finally, write out files for the conf objects
confs.each do |c|
  result = template.result(c.instance_eval("binding"))
  File.write(output_file_name % c, result)
end

tl;dr: try out C-x 5 5! (will be available in 28.1)

Emacs' frames are mighty powerful, though they can easily turn into a mess of windows.. but that's not what I wanted to share here.

Today, I wanted to open up an Info page in its own frame. That way I could keep it up and try out some command that typically mucks about with the current window setup. So, to avoid that a new frame would be handy – I thought – and so I went looking into the C-x 5 prefix's bindings:

C-x 5 C-h:

Global Bindings Starting With C-x 5:
key             binding
---             -------

C-x 5 C-f       find-file-other-frame
C-x 5 C-o       display-buffer-other-frame
C-x 5 .         xref-find-definitions-other-frame
C-x 5 0         delete-frame
C-x 5 1         delete-other-frames
C-x 5 2         make-frame-command
C-x 5 5         other-frame-prefix
C-x 5 b         switch-to-buffer-other-frame
C-x 5 d         dired-other-frame
C-x 5 f         find-file-other-frame
C-x 5 m         compose-mail-other-frame
C-x 5 o         other-frame
C-x 5 p         project-other-frame-command
C-x 5 r         find-file-read-only-other-frame

This one caught my eye – its new and coming in Emacs 28.1!

C-x 5 5         other-frame-prefix

So, to get my new frame with my desired Info page up, I hit the following keys:

C-x 5 5 M-x i n f o <RET>

That's it.

I keep up with the development branch of Emacs (mirrored on GitHub) on all my machines – and its coming features like this that remind me how much folks still care about Emacs and why I ought to keep up with the latest development efforts. Between general API improvements and the performance gains of Native Compilation, Emacs is a worthy tool to belong to every and any pragmatic fool (I dare you to be pragmatic when elisp is at your fingertips).

TIL about sysrc(8)

The sysrc tool is very handy for administering FreeBSD hosts. I've never shied away from vim /etc/rc.conf (though really emacs /etc/rc.conf for tidy editing, I'm an Emacser) so its not a tool that would have been sought out in my case. That said, any programmatic access to the system configuration is always welcomed with open arms.

Nevertheless, having more tools in the pragmatic belt makes for a better belt – as long as it makes things more practical and pragmatic in approach. In my case, I wanted to setup a tFTP server to allow me to install a fresh image on my network hardware. They require a tftp://$serverip/junos/$release_blob.tgz URL to load themselves up from and this means.. well that you need a tFTP server.

So, to set up the server we'll need have inetd start it and then start (or restart) inetd:

# Enable inetd services
sysrc inetd_enable=YES
# Configure tFTP to start (by inetd)
sed -i '' -E 's/#(tftp.*udp[[:blank:]])/\1/' /etc/inetd.conf
# Start it up!
service inetd start

My host uses ZFS as its root filesystem, so I also added a filesystem where /tftpboot lives:

zfs create -o mountpoint=/tftpboot $rootfs/tftpboot

With that, you can place files in /tftpboot that'll be retrievable by clients. Neat & simple.