oxidecomputer / oxidecomputer/omicron

test failed in CI: db-dev::test-db-dev test_db_run

Open
#7,429 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This test failed on a CI run on PR #7428:

https://github.com/oxidecomputer/omicron/pull/7428/checks?check_run_id=36364357016

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01JJSCVHZC6J3VQW664BHDCJMJ/3H7s0iynm0IsLFeoEjnGZrLNLUsKIxrfw6DW1eaibQg4GbKo/01JJSCVWF6SBW3EVQ27VPWN5SK

Excerpt from the log showing the failure:

5438	2025-01-29T17:26:22.799Z	subproc stdout: WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
5439	2025-01-29T17:26:22.807Z	subproc stdout: WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpQowpZL"
5440	2025-01-29T17:26:22.807Z	subproc stdout: 	If you would like to access the database for debugging, run the following:
5441	2025-01-29T17:26:22.807Z	subproc stdout:
5442	2025-01-29T17:26:22.807Z	subproc stdout: 	# Run the database
5443	2025-01-29T17:26:22.807Z	subproc stdout: 	cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpQowpZL/data"
5444	2025-01-29T17:26:22.807Z	subproc stdout: 	# Access the database. Note the port may change if you run multiple databases.
5445	2025-01-29T17:26:22.807Z	subproc stdout: 	cockroach sql --host=localhost:32221 --insecure
5446	2025-01-29T17:26:22.807Z	subproc stdout: Error: populating database
5447	2025-01-29T17:26:22.807Z	subproc stdout:
5448	2025-01-29T17:26:22.808Z	subproc stdout: Caused by:
5449	2025-01-29T17:26:22.808Z	subproc stdout:     0: populate
5450	2025-01-29T17:26:22.808Z	subproc stdout:     1: populating Omicron database
5451	2025-01-29T17:26:22.808Z	subproc stdout:     2: db error: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 102014976 currently allocated, 134217728 bytes in budget
5452	2025-01-29T17:26:22.808Z	subproc stdout:        HINT: Consider increasing --max-sql-memory startup parameter.
5453	2025-01-29T17:26:22.808Z	subproc stdout:        --
5454	2025-01-29T17:26:22.808Z	subproc stdout:        Try setting a higher --max-sql-memory.
5455	2025-01-29T17:26:22.808Z	subproc stdout:        --
5456	2025-01-29T17:26:22.808Z	subproc stdout:        Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
5457	2025-01-29T17:26:22.808Z	subproc stdout:        Manual inspection may be required to determine the actual state of the database.
5458	2025-01-29T17:26:22.808Z	subproc stdout:        --
5459	2025-01-29T17:26:22.808Z	subproc stdout:        See: https://github.com/cockroachdb/cockroach/issues/42061
5460	2025-01-29T17:26:22.808Z	subproc stdout:     3: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 102014976 currently allocated, 134217728 bytes in budget
5461	2025-01-29T17:26:22.808Z	subproc stdout:        HINT: Consider increasing --max-sql-memory startup parameter.
5462	2025-01-29T17:26:22.808Z	subproc stdout:        --
5463	2025-01-29T17:26:22.808Z	subproc stdout:        Try setting a higher --max-sql-memory.
5464	2025-01-29T17:26:22.808Z	subproc stdout:        --
5465	2025-01-29T17:26:22.808Z	subproc stdout:        Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
5466	2025-01-29T17:26:22.808Z	subproc stdout:        Manual inspection may be required to determine the actual state of the database.
5467	2025-01-29T17:26:22.808Z	subproc stdout:        --
5468	2025-01-29T17:26:22.808Z	subproc stdout:        See: https://github.com/cockroachdb/cockroach/issues/42061

The failure mode here looks like #7141 (but that's a different DB test so figured I'd open this for ease of searching if this recurs).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the db-dev::test-db-dev test_db_run CI check and its linked build log, focusing on the CockroachDB memory-budget error during database population. Compare the failure with issue #7141, then reproduce or investigate the test and verify that the CI database setup completes without the schema backfill failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ci-cd, databases, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.