FlowFuse / FlowFuse/flowfuse

Sort hosted instances by status with server-side pagination

Open
#7,881 0 comments 0 reactions 0 assignees View on GitHub
task
Dominant language
JavaScript
Stars
400
Forks
89
Avg merge
1d 20h
Merged PRs (30d)
149

Description

### Description

### Problem

Before we added server-side pagination to the instances table, we had a `CASE`-based SQL sort that grouped instances by state priority (errors first, then running, then stopped). That sort only covered the DB `state` column, but the effective status a user sees is actually resolved from three layers:

1. **In-flight state** (Redis cache, `project-inflightProjectState`): transitory states like `starting`, `stopping`, `restarting`, `suspending`, `installing`, `importing`, `loading`
2. **Latest driver-reported state** (Redis cache, `project-latestProjectState`): non-running states reported by the container driver (e.g. `crashed`, `safe`) that haven't been written back to the DB
3. **DB `state` column**: the persisted stable state (`running`, `suspended`, `stopped`)

The resolution priority is:
effective = inflight ?? (state === 'suspended' ? 'suspended' : (latest ?? state))

When we introduced pagination, we dropped sorting by status because it can't work correctly with `LIMIT`/`OFFSET` when the sort key (effective status) isn't fully available at the SQL level. The old `CASE` expression only looked at the DB column, so instances that are `starting` or `crashed` (per Redis) but `running` (per DB) would sort incorrectly, and pages would be inconsistent.

### What we want

Users should be able to sort their instances list by status, even with server-side pagination. The sort should reflect the *effective* status, not just the DB column.

### The challenge

The effective state lives across two systems (DB + Redis caches), so a pure SQL `ORDER BY` can't produce the right result. Some options worth considering:
* **Materialize effective state into the DB**: write the resolved state to a dedicated column (or update the existing `state` column) whenever the in-flight or driver-reported state changes. This makes SQL sorting trivial but adds write overhead and requires careful handling of cache/DB consistency.

* **Hybrid approach**: keep using the DB `state` column for the SQL sort (via `CASE` grouping like before), accept that transitory states might appear slightly out of order within a page, and let the frontend's live MQTT updates handle the visual correction. Might be "good enough" for most cases since transitory states are short-lived.

* **Fetch IDs from cache, then paginate**: resolve effective state for all instances in Redis first, sort in application code, then fetch the page of instances from the DB by ID. Scales fine for reasonable instance counts per team but could get expensive if teams have thousands.

* **Something else entirely**: open to ideas.

### Relevant code
* State resolution logic: `forge/db/controllers/Project.js` (`_publishLiveStateUnlocked`, `getInflightState`, `getLatestProjectState`)
* Cache keys: `project-inflightProjectState`, `project-latestProjectState`
* Old SQL CASE sort: `forge/db/models/Project.js` (the `orderByMostRecentFlows` block in `byTeam`)
* Pagination options: `forge/db/controllers/Project.js` (`getProjectPaginationOptions`)
* API endpoint: `forge/routes/api/team.js` (`GET /api/v1/teams/:teamId/projects`)
* Frontend list: `frontend/src/pages/team/Instances.vue`
* State definitions: `frontend/src/composables/InstanceStates.js`

### Epic/Story

_No response_

### Have you provided an initial effort estimate for this issue?

I have provided an initial effort estimate

Contributor guide

Open the contributing guide

Research direction

Start by reading state resolution in forge/db/controllers/Project.js, the orderByMostRecentFlows block in forge/db/models/Project.js, and pagination in getProjectPaginationOptions. Trace the GET /api/v1/teams/:teamId/projects endpoint and the Instances.vue consumer. Done means status sorting reflects effective Redis and database state consistently across server-paginated results.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, redis
Domain
api, backend, database, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.