oxidecomputer / oxidecomputer/omicron

test failed in CI: `db-dev::test-db-dev test_db_killed`

Open
#7,141 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This test failed on a CI run on "main":

https://github.com/oxidecomputer/omicron/runs/33299021748

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01JD6AMYY5FWJG8QEE5YV9VDMR/HviS2fo6F628FaTMfPIrWwuLVKy3mGtea6bgJKpuqOVsvMYg/01JD6AN89MD4FZ5MNX4QJ3PX8X#S5268

Excerpt from the log showing the failure:

subproc stdout: WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
subproc stdout: WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpB1d7nX"
subproc stdout: 	If you would like to access the database for debugging, run the following:
subproc stdout:
subproc stdout: 	# Run the database
subproc stdout: 	cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpB1d7nX/data"
subproc stdout: 	# Access the database. Note the port may change if you run multiple databases.
subproc stdout: 	cockroach sql --host=localhost:32221 --insecure
subproc stdout: Error: populating database
subproc stdout:
subproc stdout: Caused by:
subproc stdout:     0: populate
subproc stdout:     1: populating Omicron database
subproc stdout:     2: db error: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 101902336 currently allocated, 134217728 bytes in budget
subproc stdout:        HINT: Consider increasing --max-sql-memory startup parameter.
subproc stdout:        --
subproc stdout:        Try setting a higher --max-sql-memory.
subproc stdout:        --
subproc stdout:        Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
subproc stdout:        Manual inspection may be required to determine the actual state of the database.
subproc stdout:        --
subproc stdout:        See: https://github.com/cockroachdb/cockroach/issues/42061
subproc stdout:     3: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 101902336 currently allocated, 134217728 bytes in budget
subproc stdout:        HINT: Consider increasing --max-sql-memory startup parameter.
subproc stdout:        --
subproc stdout:        Try setting a higher --max-sql-memory.
subproc stdout:        --
subproc stdout:        Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
subproc stdout:        Manual inspection may be required to determine the actual state of the database.
subproc stdout:        --
subproc stdout:        See: https://github.com/cockroachdb/cockroach/issues/42061

--max-sql-memory is supposed to be 25% of system max memory, but this is not implemented on illumos. It appears to default to 128 MiB when system memory information is unavailable.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the db-dev::test-db-dev test_db_killed failure in the linked CI run and its detailed Buildomat log. Read how cargo xtask db-dev starts and populates CockroachDB, then investigate the illumos memory-information limitation and the reported --max-sql-memory setting. Done means the test no longer fails from the memory-budget error and does not leak the database process or temporary directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ci-cd, databases, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.