Align repository-layer DB transaction isolation levels to reduce noise from SERIALIZABLE retry errors
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Objective
Reduce meaningless SERIALIZABLE retry errors so that only meaningful 5xx errors remain visible, improving error visibility across the manager. Align repository-layer DB transaction isolation levels: use READ COMMITTED for operations that do not need serializable guarantees, and reserve SERIALIZABLE only for transactions that genuinely require it.
## Background
The DB engine default isolation level is SERIALIZABLE (models/utils.py create_async_engine default). begin_session() inherits this default, while begin_session_read_committed() overrides to READ COMMITTED. Repository code is currently mixed: ~202 begin_session() call sites vs ~101 begin_session_read_committed() call sites.
Many of these operations are idempotent or keyed upserts that do not require serializable isolation. Under SERIALIZABLE, concurrent access produces transient PostgreSQL serialization failures (SQLSTATE 40001). These are retriable-in-principle but are not consistently retried, so they surface as unhandled errors that spam the logs.
## Problem
- Meaningless SERIALIZABLE (40001) retry errors flood the manager logs.
- This noise drowns out meaningful 5xx errors that actually need attention, lowering error visibility and slowing incident triage.
## Scope
- Audit repository-layer begin_session() call sites and classify each as (a) safe to run under READ COMMITTED, or (b) genuinely requiring SERIALIZABLE.
- Migrate case (a) sites to begin_session_read_committed().
- For case (b) sites that must stay SERIALIZABLE, ensure a proper serialization-failure retry wrapper so 40001 does not escape as an unhandled error.
- Establish a clear default/guideline so new repository code picks the right isolation level.
## Acceptance Criteria
- SERIALIZABLE (SQLSTATE 40001) retry errors no longer spam the manager logs under normal operation.
- Meaningful 5xx errors are clearly visible and not buried by serialization noise.
- Each repository transaction has an intentional, documented isolation level; remaining SERIALIZABLE transactions have serialization-failure retry handling.
## Related
- BA-4112 (epic, done): Optimize database connection handling for read operations
- BA-4121 (done): Apply READ COMMITTED isolation level for database transactions
- BA-6635 (duplicate-ish bug): Container registry image rescan fails with PostgreSQL serialization error (40001) on commit
JIRA Issue: BA-6652
Contributor guide
Research direction
Start with models/utils.py to confirm the default isolation level and the begin_session_read_committed() override. Audit repository-layer begin_session() call sites, classify which can use READ COMMITTED, and identify remaining SERIALIZABLE paths needing retry handling. Done means transaction isolation is intentional and documented, serialization errors no longer flood logs, and meaningful 5xx errors remain visible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend, database
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100