oxidecomputer / oxidecomputer/dendrite

Dendrite could automatically clear power faults

Open
#71 0 comments 0 reactions 1 assignee View on GitHub

@bnaecker is already working on this.

Since Jul 14, 2025.

Dominant language
Rust
Stars
20
Forks
3
Avg merge
3d 11h
Merged PRs (30d)
4

Description

Originally reported by @bnaecker

While bringing up the next rack, we noticed that one transceiver was faulted, reporting power lost:

BRM27230053-switch # swadm sp xcvr ls
Port   State     Reset Interrupt Power  Power control Vendor           Part No.         Serial No.
qsfp0  supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65A12X
qsfp1  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X5NBW41
qsfp2  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X5QA1LJ
qsfp3  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X57BJGF
qsfp4  supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5MBDQ4
qsfp5  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X5PB5JE
qsfp6  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X72B2BL
qsfp7  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X5KBMCX
qsfp8  supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X69BVNJ
qsfp9  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X52AC5S
qsfp10 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X64A0GT
qsfp11 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X72AZ71
qsfp12 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X64A1HF
qsfp13 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X64BWFW
qsfp14 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5EBGDU
qsfp15 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5SB7J4
qsfp16 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65BPU1
qsfp17 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5DBJZT
qsfp18 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65B603
qsfp19 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5QAPG4
qsfp20 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5RA07S
qsfp21 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5PALLC
qsfp22 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65AM09
qsfp23 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65BLFZ
qsfp24 faulted (PowerLost)
qsfp25 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X72B3LF
qsfp26 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X61ABER
qsfp27 supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X5QAAJV
qsfp28 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5PABV4
qsfp29 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5QB2MY
qsfp30 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65A14C
qsfp31 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5PAQHE
BRM27230053-switch # xcvradm -i sidecar0 -t 24 vendor-info
Port Identifier               Vendor           Part             Rev  Serial           Mfg date
Some operations failed, errors below
Port Error
  24 Hardware error accessing module 24: FPGA reported an I2C error
BRM27230053-switch # xcvradm -i sidecar0 -t 24 status
Port Status
  24 PRESENT | ENABLED | LOW_POWER_MODE | INTERRUPT | FAULT_POWER_LOST
BRM27230053-switch #

@Aaron-Hartwig and I recalled that this fault is latched, so at some point we lost power and all downstream operations are destined to fail. We cleared the latch manually, and things started working fine:

BRM27230053-switch # xcvradm -i sidecar0 -t 24 clear-power-fault
BRM27230053-switch # xcvradm -i sidecar0 -t 24 status
Port Status
  24 PRESENT | ENABLED | LOW_POWER_MODE | INTERRUPT | POWER_GOOD
BRM27230053-switch #
BRM27230053-switch # xcvradm -i sidecar0 -t 24 vendor-info
Port Identifier               Vendor           Part             Rev  Serial           Mfg date
  24 QsfpPlusCmis (0x1e)      FINISAR CORP.    FTCC1112E2PCL    A    X5DA0P3          210901
BRM27230053-switch # swadm sp xcvr ls
Port   State     Reset Interrupt Power  Power control Vendor           Part No.         Serial No.
qsfp0  supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65A12X
qsfp1  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X5NBW41
qsfp2  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X5QA1LJ
qsfp3  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X57BJGF
qsfp4  supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5MBDQ4
qsfp5  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X5PB5JE
qsfp6  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X72B2BL
qsfp7  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X5KBMCX
qsfp8  supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X69BVNJ
qsfp9  supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X52AC5S
qsfp10 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X64A0GT
qsfp11 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X72AZ71
qsfp12 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X64A1HF
qsfp13 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X64BWFW
qsfp14 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5EBGDU
qsfp15 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5SB7J4
qsfp16 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65BPU1
qsfp17 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5DBJZT
qsfp18 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65B603
qsfp19 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5QAPG4
qsfp20 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5RA07S
qsfp21 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5PALLC
qsfp22 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65AM09
qsfp23 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65BLFZ
qsfp24 supported false true      low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X5DA0P3
qsfp25 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X72B3LF
qsfp26 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X61ABER
qsfp27 supported false false     low    Hardware      FINISAR CORP.    FTCC1112E2PCL    X5QAAJV
qsfp28 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5PABV4
qsfp29 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5QB2MY
qsfp30 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X65A14C
qsfp31 supported false true      high   Software      FINISAR CORP.    FTCC1112E2PCL    X5PAQHE

This is a pretty bad failure mode, because there is no way to automatically recover from this, requiring operator intervention. Even if the module's power is actually fine, it cannot work until the fault is cleared. Dendrite should consider doing this automatically. During its transceiver monitoring loop, it could note any modules with a power fault and clear that, up to some number of attempts. Should we decide to do this, it needs to be logged, and ideally we'd have a way to control this disposition through the API.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.