fm: design for health endpoint observation

As described in §1.6 RFD 589, diagnosis engines must be able to operate on information communicated both by ereports, which are edge-triggered notifications that indicate an event has occurred; and from health endpoints, which are APIs through which the current health status of a component may be requested by the control plane. We need to design the system through which diagnosis engines in the control plane may request information from these health endpoints and incorporate it into a new sitrep.

Overview of current and potential health endpoints

SP health endpoints

Health endpoint polling is particularly important in the case of diagnosis engines that deal with fault management information received from service processors, as both the lack of persistent storage on the SP and its limited amount of RAM mean that the ereport protocol is inherently lossy in the event of SP crashes or buffer overflows. However, the SP ereport protocol is designed to ensure that the control plane will always know when such data loss has occurred (see §4.3 RFD 520, §4.3.4 RFD 589), we need not poll these health endpoints on every service processor in a steady state. Instead, we will only need to request data from health endpoints as needed, either due to the receipt of a data-loss report or to continue to monitor an ongoing case.

SP health endpoints will include data requested from the existing MGS inventory, component-details, and pmbus-status APIs, as well as new interfaces that may not yet exist. Some of these are already collected by the inventory, while others are not

Host health endpoints

Presently, we collect some fault management information from the host fmd via sled-agent as part of inventory collection. This information may also be collected proactively as part of ongoing cases. We may also need to query additional existing or new sled-agent APIs for health information that is not present in the fmd APIs we currently consume.

Potential Designs

So, we need a system through which the control plane fault management system can collect health endpoint data on demand. How might that work?

This list is incomplete. You can help by expanding it.

Through the inventory

Some information from health endpoints is already collected periodically by inventory collection. One potential approach to health endpoint polling might be to just add a way for a diagnosis engine to request a new inventory collection, and then ensure any health endpoints not already included in inventory collected are added.

This is appealing on the surface, but I think it is probably not a great idea. The inventory is big, and collecting one takes a long time. The inventory collection code always tries to collect a complete inventory of the entire rack, and that's not really necessary to request the health endpoints from a single SP...and that means we will be waiting for the entire inventory to be collected just to get the one thing we actually wanted. And if we add, say, component-details for every SP component to the inventory, we're only going to make collecting one take even longer.

Therefore, I think the main way this will interact with the existing inventory code is reusing some of the implementation details and Rust types, rather than just saying all health endpoints are polled bye every inventory collection. We might also consider a way to update the inventory, or just trigger a new collection, if we see something has changed from what the latest inventory says it is.

In analysis preparation

Another option is a system where diagnosis engines can attach requests for observing a particular health endpoint to a case, and as part of the preparation phase, the fm_analysis background task goes and tries to read every health endpoint for which an observation is requested in the current sitrep. Then, the results of those observations is included as an input into the analysis phase.

The advantage of this is that it's relatively simple. It avoids having to have any kind of concurrency control for things that want to write new observations to the database, because we just don't do that, and instead only include the results of those observations in the sitrep that's generated. Making observations in the input phase means that we don't need to wait for some other process to have collected them before planning the next sitrep: if a diagnosis engine asked for an observation in a previous sitrep, we will always have tried to make that observation before the next analysis phase, inherently. It also means we don't need to go and have separate database models for every single kind of observation we might make, as they're just collected in memory whenever we are preparing inputs. While the observations and/or the conclusions made based on them would eventually be recorded durably, that would be done by having the DE attach facts to the case in the sitrep, leveraging an already-existing subsystem.

The most obvious potential disadvantage is that it's important to not block the next analysis phase for a long time. This is important because a sitrep may contain any number of disparate cases belonging to unrelated diagnosis engines, and there may be new data pertaining to some of them that requires attention, even when we aren't able to collect the data needed for another one. Analysis must always be able to proceed regardless of whether some particular DE can't get what it needs. This means that we probably shouldn't have retry loops for observations, for example. If an observation was requested, we try to read that thing once, and if it doesn't work, we just include something in the inputs that says why we weren't able to get that thing. If we wanted to retry, the diagnosis engine could leave the observation request open, so we try it again in the next sitrep.

In a separate background task

The alternative to making observations in the input phase is to instead have a separate background task which just attempts to make every observation requested in the current sitrep and write them to the database. In this design, diagnosis engines would similarly request observations attached to their cases, but the observations are made by a different task and the results are read back from the database when preparing analysis inputs. This task could either be a new fm_observation thing or part of the existing rendezvous task; personally I think it should probably be separate but I haven't thought too much about the pros and cons there yet.

The advantage here is that we have decoupled the observations themselves from the analysis task, so we can go on and produce new sitreps without potentially time consuming observations getting in the way.

On the other hand, this inherently has more moving parts than doing it in the input phase. We would need database models to represent every possible observation we might make so that we can read them in the input phase. We would probably want to do that in a way similar to the existing fm_fact tables, where one row can represent a variety of different schemas, to allow us to read them in bigger batches than in a design where each different kind of data we want to store has a separate table that might only have one or two records in it. Working with these kind of semi-strongly-typed "bag of nulls" tables can be annoying and involves more boilerplate.

In addition, this approach means we would also need to figure out how we will sort out the fact that multiple Nexii will now all be trying to satisfy the same observation requests. We would need to either ensure that they have a primary key which ensures there's exactly one observation stored for each request (which means potentially throwing away useful data until the next sitrep is written, which I don't love), or we would need some way to allow the DE to figure out which of a set of observation results is the most recent in the case where they differ (which is also inherently kind of sketchy...)