oxidecomputer / oxidecomputer/omicron
test failed in CI: `db-dev::test-db-dev test_db_killed`
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This test failed on a CI run on "main":
https://github.com/oxidecomputer/omicron/runs/33299021748
Log showing the specific test failure:
Excerpt from the log showing the failure:
subproc stdout: WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
subproc stdout: WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpB1d7nX"
subproc stdout: If you would like to access the database for debugging, run the following:
subproc stdout:
subproc stdout: # Run the database
subproc stdout: cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpB1d7nX/data"
subproc stdout: # Access the database. Note the port may change if you run multiple databases.
subproc stdout: cockroach sql --host=localhost:32221 --insecure
subproc stdout: Error: populating database
subproc stdout:
subproc stdout: Caused by:
subproc stdout: 0: populate
subproc stdout: 1: populating Omicron database
subproc stdout: 2: db error: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 101902336 currently allocated, 134217728 bytes in budget
subproc stdout: HINT: Consider increasing --max-sql-memory startup parameter.
subproc stdout: --
subproc stdout: Try setting a higher --max-sql-memory.
subproc stdout: --
subproc stdout: Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
subproc stdout: Manual inspection may be required to determine the actual state of the database.
subproc stdout: --
subproc stdout: See: https://github.com/cockroachdb/cockroach/issues/42061
subproc stdout: 3: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 101902336 currently allocated, 134217728 bytes in budget
subproc stdout: HINT: Consider increasing --max-sql-memory startup parameter.
subproc stdout: --
subproc stdout: Try setting a higher --max-sql-memory.
subproc stdout: --
subproc stdout: Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
subproc stdout: Manual inspection may be required to determine the actual state of the database.
subproc stdout: --
subproc stdout: See: https://github.com/cockroachdb/cockroach/issues/42061
--max-sql-memory is supposed to be 25% of system max memory, but this is not implemented on illumos. It appears to default to 128 MiB when system memory information is unavailable.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the db-dev::test-db-dev test_db_killed failure in the linked CI run and its detailed Buildomat log. Read how cargo xtask db-dev starts and populates CockroachDB, then investigate the illumos memory-information limitation and the reported --max-sql-memory setting. Done means the test no longer fails from the memory-budget error and does not leak the database process or temporary directory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ci-cd, databases, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100