lablup / lablup/backend.ai

Align repository-layer DB transaction isolation levels to reduce noise from SERIALIZABLE retry errors

Open
#12,465 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Objective

Reduce meaningless SERIALIZABLE retry errors so that only meaningful 5xx errors remain visible, improving error visibility across the manager. Align repository-layer DB transaction isolation levels: use READ COMMITTED for operations that do not need serializable guarantees, and reserve SERIALIZABLE only for transactions that genuinely require it.

## Background

The DB engine default isolation level is SERIALIZABLE (models/utils.py create_async_engine default). begin_session() inherits this default, while begin_session_read_committed() overrides to READ COMMITTED. Repository code is currently mixed: ~202 begin_session() call sites vs ~101 begin_session_read_committed() call sites.

Many of these operations are idempotent or keyed upserts that do not require serializable isolation. Under SERIALIZABLE, concurrent access produces transient PostgreSQL serialization failures (SQLSTATE 40001). These are retriable-in-principle but are not consistently retried, so they surface as unhandled errors that spam the logs.

## Problem

- Meaningless SERIALIZABLE (40001) retry errors flood the manager logs.
- This noise drowns out meaningful 5xx errors that actually need attention, lowering error visibility and slowing incident triage.

## Scope

- Audit repository-layer begin_session() call sites and classify each as (a) safe to run under READ COMMITTED, or (b) genuinely requiring SERIALIZABLE.
- Migrate case (a) sites to begin_session_read_committed().
- For case (b) sites that must stay SERIALIZABLE, ensure a proper serialization-failure retry wrapper so 40001 does not escape as an unhandled error.
- Establish a clear default/guideline so new repository code picks the right isolation level.

## Acceptance Criteria

- SERIALIZABLE (SQLSTATE 40001) retry errors no longer spam the manager logs under normal operation.
- Meaningful 5xx errors are clearly visible and not buried by serialization noise.
- Each repository transaction has an intentional, documented isolation level; remaining SERIALIZABLE transactions have serialization-failure retry handling.

## Related

- BA-4112 (epic, done): Optimize database connection handling for read operations
- BA-4121 (done): Apply READ COMMITTED isolation level for database transactions
- BA-6635 (duplicate-ish bug): Container registry image rescan fails with PostgreSQL serialization error (40001) on commit

JIRA Issue: BA-6652

Contributor guide

Open the contributing guide

Research direction

Start with models/utils.py to confirm the default isolation level and the begin_session_read_committed() override. Audit repository-layer begin_session() call sites, classify which can use READ COMMITTED, and identify remaining SERIALIZABLE paths needing retry handling. Done means transaction isolation is intentional and documented, serialization errors no longer flood logs, and meaningful 5xx errors remain visible.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, database
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.