Sort hosted instances by status with server-side pagination
- Dominant language
- JavaScript
- Stars
- 400
- Forks
- 89
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 149
Description
### Description
### Problem
Before we added server-side pagination to the instances table, we had a `CASE`-based SQL sort that grouped instances by state priority (errors first, then running, then stopped). That sort only covered the DB `state` column, but the effective status a user sees is actually resolved from three layers:
1. **In-flight state** (Redis cache, `project-inflightProjectState`): transitory states like `starting`, `stopping`, `restarting`, `suspending`, `installing`, `importing`, `loading`
2. **Latest driver-reported state** (Redis cache, `project-latestProjectState`): non-running states reported by the container driver (e.g. `crashed`, `safe`) that haven't been written back to the DB
3. **DB `state` column**: the persisted stable state (`running`, `suspended`, `stopped`)
The resolution priority is:
effective = inflight ?? (state === 'suspended' ? 'suspended' : (latest ?? state))
When we introduced pagination, we dropped sorting by status because it can't work correctly with `LIMIT`/`OFFSET` when the sort key (effective status) isn't fully available at the SQL level. The old `CASE` expression only looked at the DB column, so instances that are `starting` or `crashed` (per Redis) but `running` (per DB) would sort incorrectly, and pages would be inconsistent.
### What we want
Users should be able to sort their instances list by status, even with server-side pagination. The sort should reflect the *effective* status, not just the DB column.
### The challenge
The effective state lives across two systems (DB + Redis caches), so a pure SQL `ORDER BY` can't produce the right result. Some options worth considering:
* **Materialize effective state into the DB**: write the resolved state to a dedicated column (or update the existing `state` column) whenever the in-flight or driver-reported state changes. This makes SQL sorting trivial but adds write overhead and requires careful handling of cache/DB consistency.
* **Hybrid approach**: keep using the DB `state` column for the SQL sort (via `CASE` grouping like before), accept that transitory states might appear slightly out of order within a page, and let the frontend's live MQTT updates handle the visual correction. Might be "good enough" for most cases since transitory states are short-lived.
* **Fetch IDs from cache, then paginate**: resolve effective state for all instances in Redis first, sort in application code, then fetch the page of instances from the DB by ID. Scales fine for reasonable instance counts per team but could get expensive if teams have thousands.
* **Something else entirely**: open to ideas.
### Relevant code
* State resolution logic: `forge/db/controllers/Project.js` (`_publishLiveStateUnlocked`, `getInflightState`, `getLatestProjectState`)
* Cache keys: `project-inflightProjectState`, `project-latestProjectState`
* Old SQL CASE sort: `forge/db/models/Project.js` (the `orderByMostRecentFlows` block in `byTeam`)
* Pagination options: `forge/db/controllers/Project.js` (`getProjectPaginationOptions`)
* API endpoint: `forge/routes/api/team.js` (`GET /api/v1/teams/:teamId/projects`)
* Frontend list: `frontend/src/pages/team/Instances.vue`
* State definitions: `frontend/src/composables/InstanceStates.js`
### Epic/Story
_No response_
### Have you provided an initial effort estimate for this issue?
I have provided an initial effort estimate
Contributor guide
Research direction
Start by reading state resolution in forge/db/controllers/Project.js, the orderByMostRecentFlows block in forge/db/models/Project.js, and pagination in getProjectPaginationOptions. Trace the GET /api/v1/teams/:teamId/projects endpoint and the Instances.vue consumer. Done means status sorting reflects effective Redis and database state consistently across server-paginated results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, redis
- Domain
- api, backend, database, frontend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100