oxidecomputer / oxidecomputer/dendrite

Switch zone space exhausted from logs

Open
#266 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
20
Forks
3
Avg merge
3d 11h
Merged PRs (30d)
4

Description

We found a switch zone that looked like this

# zfs list rpool/zone/oxz_switch
NAME                    USED  AVAIL     REFER  MOUNTPOINT
rpool/zone/oxz_switch  2.97G  4.82M     2.97G  /zone/oxz_switch

due to this

-rw-r--r--   1 root     root     1703936 May  8 07:30 oxide-dendrite:default.log.0
drwxr-xr-x   2 root     sys          132 May  8 07:30 .
-rw-r--r--   1 root     root      393216 May  8 07:15 oxide-dendrite:default.log.1
-rw-r--r--   1 root     root     1441792 May  8 07:00 oxide-dendrite:default.log.2
-rw-r--r--   1 root     root     1383575 May  8 06:47 oxide-mg-ddm:default.log
-rw-r--r--   1 root     root     14286848 May  8 06:45 oxide-dendrite:default.log.3
-rw-r--r--   1 root     root     29766746 May  8 06:45 oxide-wicketd:default.log
-rw-r--r--   1 root     root      289898 May  8 06:45 oxide-pumpkind:default.log
-rw-r--r--   1 root     root     3612606464 May  8 06:31 oxide-dendrite:default.log.4
-rw-r--r--   1 root     root     4185253382 May  8 06:31 oxide-dendrite:default.log
-rw-r--r--   1 root     root     80839571 May  8 06:31 oxide-mgd:default.log
-rw-r--r--   1 root     root     61343803 May  8 06:31 oxide-mgs:default.log
-rw-r--r--   1 root     root     40323877 May  8 06:31 oxide-tfport:default.log
-rw-r--r--   1 root     root     2709568 May  8 06:26 oxide-lldpd:default.log
-rw-r--r--   1 root     root     4195630867 May  8 06:15 oxide-dendrite:default.log.5
-rw-r--r--   1 root     root     4203580961 May  8 06:00 oxide-dendrite:default.log.6
-rw-r--r--   1 root     root     4194655483 May  8 05:45 oxide-dendrite:default.log.7
-rw-r--r--   1 root     root     4196811310 May  8 05:30 oxide-dendrite:default.log.8
-rw-r--r--   1 root     root     4177313624 May  8 05:15 oxide-dendrite:default.log.9
-rw-r--r--   1 root     root     115733711 May  8 05:15 oxide-mgs:default.log.0
-rw-r--r--   1 root     root     107470637 May  8 05:00 oxide-mgd:default.log.0
-rw-r--r--   1 root     root     4191302754 May  8 05:00 oxide-dendrite:default.log.10
-rw-r--r--   1 root     root     4188813289 May  8 04:45 oxide-dendrite:default.log.11
-rw-r--r--   1 root     root     4222101031 May  8 04:30 oxide-dendrite:default.log.12

A sample log file can be found at /staff/core/dendrite-226. Due to production urgency we needed to move quickly with this one so that is all I have at the moment.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the sample log at /staff/core/dendrite-226 and tracing how switch-zone service logs are configured and rotated. Done should mean log growth is bounded so a switch zone cannot be exhausted by service logs; the issue does not name source files or tests to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.