HarperFast / HarperFast/harper
Component-created system tables are unreadable by any caller, and report as 'does not exist'
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Five tables in the `system` database cannot be read through the operations API by **any** caller, including `super_user`. Every read operation fails with HTTP 403 and the message `Table 'system.' does not exist` — while `describe_database system` enumerates those same tables in the same session.
The affected tables are all created lazily at runtime by their owning component. Tables that exist at startup are unaffected.
Observed on **Harper 5.2.0**.
## Affected tables
Probed all 23 tables in the `system` database of a running 5.2.0 cluster with three read operations each (`describe_table` with `skip_record_count`, `sql SELECT … LIMIT 1`, and `search_by_hash` with a non-existent key):
| gated — all three fail | owning component |
|---|---|
| `hdb_scheduler_state` | scheduler engine (`resources/scheduler/engine.ts`) |
| `hdb_model_calls` | models (`resources/models/analyticsTable.ts`) |
| `hdb_session` | auth (`security/auth.ts`) |
| `hdb_status` | node status (`server/status/index.ts`) |
| `hdb_waf_rules` | WAF |
The other 18 — including `hdb_user`, `hdb_role`, `hdb_nodes`, `hdb_secret`, `hdb_analytics`, `hdb_job`, `hdb_info`, `hdb_certificate`, `hdb_durable_session` — are fully readable.
Failures are deterministic: 0/16 successes on repeat for a gated table, 16/16 for the control.
## Reproduce
Against any 5.2.0 instance whose scheduler has registered at least one job, authenticated as a `super_user`:
```jsonc
// 200 — the table is listed in the response
{ "operation": "describe_database", "database": "system" }
// 403 for each of these
{ "operation": "describe_table", "database": "system", "table": "hdb_scheduler_state", "skip_record_count": true }
{ "operation": "search_by_hash", "database": "system", "table": "hdb_scheduler_state", "hash_values": ["leader"], "get_attributes": ["*"] }
{ "operation": "search_by_value", "database": "system", "table": "hdb_scheduler_state", "search_attribute": "id", "search_value": "*" }
{ "operation": "search_by_conditions", "database": "system", "table": "hdb_scheduler_state", "operator": "and", "conditions": [ /* … */ ] }
{ "operation": "sql", "sql": "SELECT * FROM system.hdb_scheduler_state LIMIT 1" }
```
Response body in every failing case:
```json
{
"error": "This operation is not authorized due to role restrictions and/or invalid database items",
"unauthorized_access": [],
"invalid_schema_items": ["Table 'system.hdb_scheduler_state' does not exist"]
}
```
`cluster_status` returns 200 on the same credentials in the same session, so this is not an authentication or connectivity problem.
## Expected
Either the table is readable by a `super_user`, or the refusal says so. `does not exist` for a table the API itself enumerates is misleading in both directions.
## Analysis
`unauthorized_access` is empty and the message is `TABLE_NOT_FOUND`, which identifies the branch: in `utility/operation_authorization.ts`, `hasPermissions()` emits `SCHEMA_NOT_FOUND` at ~line 753 when `userPerms[schema]` is missing, and `TABLE_NOT_FOUND` at ~line 771 when the schema entry exists but the table is absent from `userPerms[schema].tables`.
Since the error is `TABLE_NOT_FOUND`, the expanded role carries a `system` entry with a `.tables` map, and the five tables are simply **not in it**. The eighteen that resolve are present.
This looks like an ordering problem: the per-table permission map is built from the schema as it exists at role-resolution time, whereas these five are created later by `table({ database: 'system', … })` when their component first needs them — for the scheduler, `getStateTable()` is only reached after `startSchedulerEngine()`, itself gated on `getWorkerIndex() === 0`. `describe_database` escapes it because it carries no table in its permission map and enumerates the live registry instead, which is why the table is visible but unaddressable.
Note that `super_user` deliberately does not receive the blanket bypass for system-schema operations (`if (isSuperUser && !isSuSystemOperation) return null`), so there is no path around the per-table lookup for these.
### Ruled out
- **A denylist.** None exists; none of the five is named anywhere outside its own subsystem.
- **`NON_REPLICATING_SYSTEM_TABLES`.** Contains `hdb_temp`, `hdb_job`, `hdb_info` — all accessible — and omits four of the five gated tables.
- **`SYSTEM_TABLE_NAMES`.** 12 entries, but 18 tables are accessible.
- **Role configuration.** `user_info` reports `{ super_user: true }`; `unauthorized_access` is empty in every response.
- **Worker-thread routing.** Deterministic across 16 repeats in both directions.
## Impact
Harper's own scheduler state is unobservable through the operations API. Any tooling that wants to confirm the native scheduler is healthy — leader lease, last run, last status per job — has no supported way to read it remotely, and the error message actively points investigators at the wrong conclusion. It cost us a debugging cycle chasing a supposed renamed-table theory before we traced the authorization path; `hdb_scheduler_state` is current in 5.2.0 and referenced in `resources/scheduler/engine.ts`.
The same applies to `hdb_status` and `hdb_session`, which are natural targets for operational monitoring.
## Suggested directions
1. Register component-created system tables in the permission map when they are created, rather than only at role-resolution time.
2. Failing that, allow `super_user` reads of any table present in the live registry for the `system` database.
3. Independently: distinguish "table not found" from "not permitted" in the response. The current message is wrong whenever the table demonstrably exists.
Issue generated by kAIle (Claude Opus 5)
Contributor guide
Assessment
This issue has not been assessed yet.