Document minimum RAM requirement and recover from dirty migrations when TimescaleDB OOMs
- Dominant language
- Go
- Stars
- 55
- Forks
- 16
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 84
Description
### Describe the bug
With the Proxmox default of 512MB RAM, TimescaleDB fails to start. Migrations partially apply, leaving the schema in a dirty state, and `fleet-api` then logs `failed to run migrations: Dirty database version 16. Fix and force version.` with no in-product recovery path. Raising the VM to 4GB allows TimescaleDB to boot, but the schema is already wedged on that run.
Two asks in one issue since they share the same root cause (low-resource hosts):
1. **Docs:** add a minimum RAM requirement (4GB) to the deployment docs.
2. **Server:** harden migration startup so a transient DB failure does not silently wedge the schema.
### Preconditions
- Host/VM with ~512MB RAM (Proxmox default)
- TimescaleDB via the provided docker-compose stack
### Steps to reproduce
1. Provision a Proxmox VM with the default 512MB RAM.
2. Install Proto Fleet per [`deployment-files/README.md`](https://github.com/block/proto-fleet/blob/main/deployment-files/README.md).
3. Observe `fleet-api` failing with `Dirty database version 16. Fix and force version.`.
4. Raise the VM to 4GB and restart. TimescaleDB boots, but the schema is still dirty from the previous partial migration.
### Expected behavior
**Docs:** add a "System requirements" section to [`deployment-files/README.md`](https://github.com/block/proto-fleet/blob/main/deployment-files/README.md) specifying a 4GB RAM minimum. TimescaleDB tuned at [`server/docker-compose.base.yaml:37-72`](https://github.com/block/proto-fleet/blob/main/server/docker-compose.base.yaml#L37-L72) (`effective_cache_size=768MB`, `shared_buffers=256MB`) needs headroom above 512MB.
**Server:** `runMigrations` at [`server/internal/infrastructure/db/database_connection.go:117-146`](https://github.com/block/proto-fleet/blob/main/server/internal/infrastructure/db/database_connection.go#L117-L146) reads `dirty` from `m.Version()` but only logs it at INFO (lines 139-143). Either:
- fail startup with a clear remediation message when `dirty` is true, or
- add a bounded retry on `m.Up()` when the underlying connection flaps, so transient DB restarts on low-resource hosts do not leave the schema wedged.
Test utilities at [`server/internal/testutil/database_setup.go:59-63`](https://github.com/block/proto-fleet/blob/main/server/internal/testutil/database_setup.go#L59-L63) already drop-and-recreate to dodge this. Production code has no equivalent guard.
### Proto Fleet version
v0.2.6
### Environment
- Host: Proxmox VM
- RAM: 512MB (default), raised to 4GB to recover
### Logs
```text
fleet-api-1 | time=2026-04-24T18:26:14.671Z level=INFO msg=\"connected to database\" address=timescaledb:5432 database=fleet
fleet-api-1 | time=2026-04-24T18:26:14.693Z level=ERROR msg=\"failed to run migrations: failed to run migrations: Dirty database version 16. Fix and force version.\"
```
Contributor guide
Research direction
Start with deployment-files/README.md for the deployment requirements, then read server/docker-compose.base.yaml lines 37-72 and runMigrations in server/internal/infrastructure/db/database_connection.go lines 117-146. Review the existing database setup in server/internal/testutil/database_setup.go lines 59-63 and the migration failure behavior. Done means the 4GB minimum is documented and startup handles dirty or transient migration failures with a clear, tested outcome.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, go, postgresql
- Domain
- database, devops, documentation
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100