debug dataset could be better organized

(This should maybe be an RFD, but I don't plan to do anything with this now, and just wanted to write it down for future-us.)

Problem

On each U.2 zpool, in the encrypted part, we have a debug dataset (/pool/ext/$uuid/crypt/debug) that we use for long-term storage of kernel crash dumps, process core dumps, rotated SMF log files, and support bundles. But the structure is a bit haphazard today.

In the root of the debug dataset, we have things like:

• core.oxz_switch.mgadm.28311.1762901571: a process core dump
• unix.0: a kernel crash dump
• bounds: an artifact of having written out a kernel crash dump
• oxz_cockroachdb_8bbea076-ff60-4330-8302-383e18140ef3: a directory containing all of the archived log files for the zone called oxz_cockroachdb_8bbea076-ff60-4330-8302-383e18140ef3. (Is there anything else that currently goes in this directory?)
• 22103c5a-a28f-404a-84a6-3235eb0f7335: a directory containing a support bundle (note: this is actually the mountpoint of a per-support-bundle dataset)

Archived log files go into those per-zone directories.

I have personally found it a little hard both to figure out:

• given a thing that's there (like a support bundle directory), what is it? (I think there's no way to discover the answer to this -- you need to know the above heuristics.)
• given a thing I want to find (like a log file), where is it? (Same.)
• given a thing I want to add, where should it go?

This came up for me because of this last question. With #7278, we'll be archiving Reconfigurator state files, which at this layer are just "arbitrary debug data from zone X". If that goes into oxz_cockroachdb_8bbea076-ff60-4330-8302-383e18140ef3, then it's hard to distinguish from a log file. So I could put it in a new top-level directory like debug_drop/$zonename, but then we have two different directories with per-zone data in them.

Suggestion

It might be nice if the directory structure were more like:

/pool/ext/*/crypt/debug/zone/$zone/{logs,dropbox}
/pool/ext/*/crypt/debug/core (or this could go under the zone)
/pool/ext/*/crypt/debug/crash
/pool/ext/*/crypt/debug/support-bundles

This seems more discoverable and extensible. And if we built more automation atop this stuff, it wouldn't have to encode complicated rules involving parsing filenames. If it wanted to grab all the core files, for example, it would know exactly what directory/directories to look in.

Migration

Say we wanted to move to that. We could:

• Call the existing layout v1 and the new layout v2.
• Create a metadata.json file that goes in the root of the dataset that just contains the version. This tells consumers whether the dataset uses the old or new directory structure.
• Migrate from v1 to v2 mostly by creating the directories we want and then moving files/directories around. This should not require copying data so it should be fairly quick, though it's not atomic so we'd need to deal with crashing in the middle.

The bigger challenge would probably be migration of consumers, which are:

• people
• tooling like oxlog
• automation like support bundle assembly

For this I'd propose that we create a crate and CLI tool called oxd for this. This is basically an extension of oxlog for all the data here.

# List kernel crash dumps
oxd ls crash

# List process core dumps
oxd ls core

oxd would handle both versions of the directory structure. Then we could:

1. Implement this crate and CLI tool.
2. Change existing automation (at least oxlog and support bundle assembly) to use oxd.
3. Implement the physical migration plan above, knowing we won't break anything.

---

There's another piece of this, which is that with each file we also have a bunch of metadata:

• The filename for core files includes the zonename, executable name, pid, and a timestamp. Some of this information (and other useful information, like the SMF FMRI) is also in the core file.
• The path for SMF log files also includes the zonename, SMF service name, and a timestamp.

It would probably be really useful to be able to query on a bunch of this information, to be able to do things like:

# List Nexus core files
oxd ls core --smf-service=nexus

# List all recent crash dumps
oxd ls crash --since 7d

The next step is to want to go further and be able to query this across all sleds.

(This has a lot in common with thoth.)

I'm not really sure where to go with this. I think it would make sense to organize the directory better and address the metadata problem separately, maybe with a small indexing/querying subsystem.

---

@jgallagher @smklein and I discussed this briefly on the 2025-11-14 update watercooler.

Start by examining the existing oxlog behavior and support bundle assembly, especially how they discover debug data in the current dataset layout. Compare those consumers with the proposed v1/v2 structure and the #7278 Reconfigurator state-file use case. Done would require a decided scope, layout, migration approach, and consumer changes; this issue currently records possibilities rather than a bounded task.