dbcli / dbcli/symphony-dbcli

Make Alembic the canonical SQLite migration path

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Goal

Make database schema changes predictable during VM deploys by moving schema ownership to Alembic and making deployment/startup run the migration path explicitly.

The repo already has Alembic scaffolding (`alembic.ini`, `migrations/env.py`, and `migrations/versions/0001` through `0007`), but the current runtime path still relies on custom startup repair code:

- `symphony-dbcli init-db` calls `Store.init()` and `create_model_tables(...)`.
- FastAPI startup also calls `Store.init()` and `create_model_tables(...)`.
- `Store.init()` creates raw-SQL legacy tables and runs `_migrate(...)` for additive column repairs.
- `create_model_tables(...)` calls SQLAlchemy `Base.metadata.create_all(...)` and then `repair_sqlite_model_tables(...)`.
- `scripts/provision-exedev-vm.sh` runs `uv run symphony-dbcli --profile prod init-db` before installing/restarting the service, but the generated systemd unit does not run migrations on every restart.

We should make `init-db` the single blessed migration command, backed by Alembic, and then have the VM service run it safely before `serve` starts.

## Non-goals

- Do not rewrite all `Store` data access to SQLAlchemy ORM as part of this issue.
- Do not remove the existing SQLite DB or require manual data export/import.
- Do not change the configured DB location.
- Do not mix unrelated dashboard/runtime behavior changes into the migration work.

## Phase 1: Schema inventory and migration plan

Produce a short implementation note or PR description that maps every existing schema owner to the future Alembic owner.

Inventory these areas:

- Raw SQLite tables in `src/symphony_dbcli/store.py` `SCHEMA`.
- Legacy repairs in `src/symphony_dbcli/store.py` `_migrate(...)`.
- SQLAlchemy models in `src/symphony_dbcli/models.py`.
- SQLAlchemy repair entries in `_SQLITE_COLUMN_REPAIRS`.
- Existing Alembic migrations in `migrations/versions`.
- The `init-db` path in `src/symphony_dbcli/cli.py`.
- The FastAPI startup path in `src/symphony_dbcli/web/app.py`.
- The generated systemd unit in `scripts/provision-exedev-vm.sh`.

Progress checkpoint:

- A reviewer can tell which tables are already covered by Alembic and which tables still depend on `Store.init()` / repair logic.
- The plan states whether each existing custom repair will be kept temporarily, moved into Alembic, or retired.

## Phase 2: Make Alembic runnable from the app config

Make Alembic use the same database path as `symphony-dbcli --profile ... init-db`.

Expected work:

- Add a small internal migration runner or helper that resolves `WorkflowConfig.database.path` and runs `alembic upgrade head` against that SQLite DB.
- Avoid requiring operators to manually construct `SYMPHONY_DATABASE_URL`.
- Keep this helper usable from tests without shelling out when practical.
- Ensure the migration command is idempotent.

Progress checkpoint:

- A test can run Alembic against a temporary empty SQLite DB using the same config path resolution that production uses.
- Running the migration command twice succeeds without changing data or failing.

## Phase 3: Move SQLAlchemy-managed table changes fully under Alembic

Bring the SQLAlchemy model tables into a migration-first workflow.

Expected work:

- Verify the current Alembic head creates/updates the SQLAlchemy-managed tables to match the current models.
- Move any active `_SQLITE_COLUMN_REPAIRS` entries for model tables into Alembic revisions where they are still needed.
- Add tests that compare the schema from a fresh Alembic upgrade with the expected model columns/indexes.
- Decide how to handle existing DBs that may already have columns added by repair code before Alembic knows about them.

Progress checkpoint:

- Fresh DB + `alembic upgrade head` yields the expected source/work-item/chat schema.
- Existing DBs with previously repaired columns can still upgrade cleanly.

## Phase 4: Cover legacy `Store` tables with handwritten Alembic migrations

Alembic does not require ORM models, so keep `Store` as the data access layer for now and add handwritten migrations for its raw-SQL tables.

Expected work:

- Create Alembic revisions for the legacy tables currently created by `Store.SCHEMA`.
- Create Alembic revisions for the additive repairs currently in `Store._migrate(...)`.
- Preserve existing data when upgrading a populated DB.
- Add a test fixture for an older/minimal DB and verify upgrade to head adds missing columns.

Progress checkpoint:

- A fresh DB migrated with Alembic has both the legacy `Store` tables and SQLAlchemy-managed tables.
- A legacy DB missing columns such as worker heartbeat/deadline or PR cleanup columns upgrades to head successfully.
- No broad `Store` ORM rewrite is required.

## Phase 5: Make `init-db` the migration facade

Change `symphony-dbcli init-db` so operators use one command for schema setup and upgrade.

Expected work:

- Have `init-db` run Alembic migrations to head.
- Keep any required transitional compatibility behavior only where necessary and clearly mark it as temporary.
- Ensure workflow version loading still happens after schema is ready.
- Return clear errors if migration fails.

Progress checkpoint:

- `uv run symphony-dbcli --profile prod init-db` upgrades the configured DB to the latest schema.
- The command remains safe to run repeatedly.
- Tests cover empty DB, current DB, and at least one older DB shape.

## Phase 6: Update VM provisioning and systemd startup behavior

Make deploy/restart apply migrations before the app starts.

Expected work:

- Update `scripts/provision-exedev-vm.sh` so the generated systemd unit runs the migration facade before `serve`.
- Prefer `ExecStartPre= run symphony-dbcli --profile prod init-db` if it behaves cleanly under systemd.
- Keep `ExecStart= run symphony-dbcli --profile prod serve --no-reload` focused on serving/runtime work.
- Make failure behavior obvious: if migration fails, the service should not start with a mismatched schema.

Progress checkpoint:

- Re-running the provisioning script installs a unit that migrates on start/restart.
- `systemctl restart symphony-dbcli.service` applies pending migrations before launching the app.
- Failed migrations leave useful logs in `journalctl`.

## Phase 7: Retire duplicate custom repair paths

After Alembic owns the schema, remove or narrow the old repair mechanisms.

Expected work:

- Remove redundant `_migrate(...)` column additions that Alembic now owns, or keep only a guarded compatibility shim with a documented removal plan.
- Remove redundant `_SQLITE_COLUMN_REPAIRS` entries where Alembic now handles the change.
- Avoid `Base.metadata.create_all(...)` silently papering over missing migrations in production paths, unless intentionally kept for tests only.
- Update docs so future schema changes require Alembic migrations.

Progress checkpoint:

- New DB columns are introduced through Alembic revisions, not hidden startup repair lists.
- Documentation tells future contributors where migrations live and how to test them.
- Quality gates pass: `uv run ruff check .`, `uv run ruff format --check .`, `uv run mypy`, and `uv run pytest`.

## Suggested validation matrix

- Empty SQLite DB -> `init-db` -> app starts.
- Existing local DB -> `init-db` -> app starts and data remains readable.
- Older schema fixture -> `init-db` -> missing columns added and data preserved.
- VM-style path/profile -> `init-db` -> service start succeeds.
- Running `init-db` twice is a no-op.
- Migration failure prevents service start and logs a clear error.

## Open questions

- Should FastAPI startup keep running migrations directly, or should production rely on `ExecStartPre` and fail fast if schema is stale?
- Should legacy `Store` tables be represented as SQLAlchemy Core `Table` metadata for schema clarity, without rewriting `Store` methods?
- Do we want a one-time Alembic `stamp` path for existing databases that have already been repaired by startup code?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.