oxidecomputer / oxidecomputer/omicron
test failed in CI: db-dev::test-db-dev test_db_run
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This test failed on a CI run on PR #7428:
https://github.com/oxidecomputer/omicron/pull/7428/checks?check_run_id=36364357016
Log showing the specific test failure:
Excerpt from the log showing the failure:
5438 2025-01-29T17:26:22.799Z subproc stdout: WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
5439 2025-01-29T17:26:22.807Z subproc stdout: WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpQowpZL"
5440 2025-01-29T17:26:22.807Z subproc stdout: If you would like to access the database for debugging, run the following:
5441 2025-01-29T17:26:22.807Z subproc stdout:
5442 2025-01-29T17:26:22.807Z subproc stdout: # Run the database
5443 2025-01-29T17:26:22.807Z subproc stdout: cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpQowpZL/data"
5444 2025-01-29T17:26:22.807Z subproc stdout: # Access the database. Note the port may change if you run multiple databases.
5445 2025-01-29T17:26:22.807Z subproc stdout: cockroach sql --host=localhost:32221 --insecure
5446 2025-01-29T17:26:22.807Z subproc stdout: Error: populating database
5447 2025-01-29T17:26:22.807Z subproc stdout:
5448 2025-01-29T17:26:22.808Z subproc stdout: Caused by:
5449 2025-01-29T17:26:22.808Z subproc stdout: 0: populate
5450 2025-01-29T17:26:22.808Z subproc stdout: 1: populating Omicron database
5451 2025-01-29T17:26:22.808Z subproc stdout: 2: db error: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 102014976 currently allocated, 134217728 bytes in budget
5452 2025-01-29T17:26:22.808Z subproc stdout: HINT: Consider increasing --max-sql-memory startup parameter.
5453 2025-01-29T17:26:22.808Z subproc stdout: --
5454 2025-01-29T17:26:22.808Z subproc stdout: Try setting a higher --max-sql-memory.
5455 2025-01-29T17:26:22.808Z subproc stdout: --
5456 2025-01-29T17:26:22.808Z subproc stdout: Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
5457 2025-01-29T17:26:22.808Z subproc stdout: Manual inspection may be required to determine the actual state of the database.
5458 2025-01-29T17:26:22.808Z subproc stdout: --
5459 2025-01-29T17:26:22.808Z subproc stdout: See: https://github.com/cockroachdb/cockroach/issues/42061
5460 2025-01-29T17:26:22.808Z subproc stdout: 3: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 102014976 currently allocated, 134217728 bytes in budget
5461 2025-01-29T17:26:22.808Z subproc stdout: HINT: Consider increasing --max-sql-memory startup parameter.
5462 2025-01-29T17:26:22.808Z subproc stdout: --
5463 2025-01-29T17:26:22.808Z subproc stdout: Try setting a higher --max-sql-memory.
5464 2025-01-29T17:26:22.808Z subproc stdout: --
5465 2025-01-29T17:26:22.808Z subproc stdout: Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
5466 2025-01-29T17:26:22.808Z subproc stdout: Manual inspection may be required to determine the actual state of the database.
5467 2025-01-29T17:26:22.808Z subproc stdout: --
5468 2025-01-29T17:26:22.808Z subproc stdout: See: https://github.com/cockroachdb/cockroach/issues/42061
The failure mode here looks like #7141 (but that's a different DB test so figured I'd open this for ease of searching if this recurs).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the db-dev::test-db-dev test_db_run CI check and its linked build log, focusing on the CockroachDB memory-budget error during database population. Compare the failure with issue #7141, then reproduce or investigate the test and verify that the CI database setup completes without the schema backfill failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ci-cd, databases, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100