Repairing Nix store

I have seen Nix break itself (its DB) a few times. Once I even managed to wipe 90% of my /nix/store after not noticing the error and running nix-collect-garbage -d (to clear space).

Nix uses a database to track added paths and facilitate untrusted user's added paths with respect to the derivations used to create them. If this database is corrupted, then Nix has no idea about the paths that already exist on disk. In the case where I wiped a bunch of paths off my disk (deleting basically the entire system, the equivalent to rm -rf /usr on many other Linux distributions) the paths aren't at all registered, tracked, or considered valid.

So, Nix says “dunno what these are” and carries on. If you garbage collect during this time, well, those paths aren't registered and suspect to deletion.

You probably came for the repair. On to that bit now.

When needing to repair the database you'll need to either do one of two things:

  1. mangle whatever you have left of the database
  2. call it a loss and create a new database

I've done 1. It isn't fun. You'll need to check that the SQLite3 structures are valid and that constraints are still adhered to – this amounts to first running sqlite3 /nix/var/nix/db/db.sqlite "pragma integrity_check" to verify, and then dumping the database to individual insertions to remove the culprit (if you can find them all, which you can do, but holy heck is that tedious).

# dump whatever the database contains, to edit and create a new db with
sqlite3 -readonly /nix/var/nix/db/db.sqlite .dump > /nix/var/nix/db/dump-$EPOCHSECONDS.sql

# after editing, load it into a new database.. and cross your fingers
mv /nix/var/nix/db/{db.sqlite,db.sqlite.$EPOCHSECONDS}
sqlite3 /nix/var/nix/db/db.sqlite < /nix/var/nix/db/edited-dump.sql

Note: this is not guaranteed to work. Invalid paths and database constraints may plague the process over many iterations. You've been warned :)

After having done 1 more than once (yes, more than once), I've found its not worthwhile for my workstation purposes. In practice, the actual content is still on disk and won't be deleted if you're being cautious. Route 2 keeps these contents around – during which time I'd even bet the store may even still serve its purposes for you – and you instead import the paths into a fresh database. This does mean you'll lose some amount of information. I suspect that this means you'll lose the association between a store path, the producing derivation, and possibly compressed logs. I haven't confirmed this myself because I'm content gaining operability rather than worry about lost logs for successfully built derivations.

How do you go about taking route 2?

I'll show you, but first there's some amount of care to exercise:

# stash the bad db away. Just in case.
mkdir -p /nix/var/nix/db/prev.$$
mv /nix/var/nix/db/{schema,reserved,db.sqlite} /nix/var/nix/db/prev.$$

If you don't have nix on PATH still, then you might find it in /nix/store (or get a new copy from https://nixos.org/nix):

ls -1 /nix/store/*nix*/bin/nix

Pick one. If you can recall which version you were using – use that! – if not, then use one of them (preferably not a pre-release version – which might be why you landed here in the first place).

/nix/store/n8x6ig1yf8ffpa07mwvxg6b7ilrrvfy1-nix-2.4/bin/nix-store --init

And then, again, there's tedium or there's pragmatic ignorance (reasonable laziness perhaps?), either to live with unregistered paths (that are registered on demand as you use Nix) or to add the paths back to the database.

Again, I've done both. On a big builder host I'd take the time to import the paths, but on my workstation I'd rather just move on with my life. It isn't worth the time in that case because you will grow the database back to the learn about what you still have on disk. If the derivation continues to produce the same hash then Nix can and will import paths. Not in all cases, mind you, but still. It's a cost that some regions of the world with limited internet can't accept so I won't say its the right choice for everyone.