HarperFast / HarperFast/harper

Component-created system tables are unreadable by any caller, and report as 'does not exist'

Open
#2,120 0 comments 0 reactions 1 assignee Claimed by @kylebernhardy View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

Five tables in the `system` database cannot be read through the operations API by **any** caller, including `super_user`. Every read operation fails with HTTP 403 and the message `Table 'system.' does not exist` — while `describe_database system` enumerates those same tables in the same session.

The affected tables are all created lazily at runtime by their owning component. Tables that exist at startup are unaffected.

Observed on **Harper 5.2.0**.

## Affected tables

Probed all 23 tables in the `system` database of a running 5.2.0 cluster with three read operations each (`describe_table` with `skip_record_count`, `sql SELECT … LIMIT 1`, and `search_by_hash` with a non-existent key):

| gated — all three fail | owning component |
|---|---|
| `hdb_scheduler_state` | scheduler engine (`resources/scheduler/engine.ts`) |
| `hdb_model_calls` | models (`resources/models/analyticsTable.ts`) |
| `hdb_session` | auth (`security/auth.ts`) |
| `hdb_status` | node status (`server/status/index.ts`) |
| `hdb_waf_rules` | WAF |

The other 18 — including `hdb_user`, `hdb_role`, `hdb_nodes`, `hdb_secret`, `hdb_analytics`, `hdb_job`, `hdb_info`, `hdb_certificate`, `hdb_durable_session` — are fully readable.

Failures are deterministic: 0/16 successes on repeat for a gated table, 16/16 for the control.

## Reproduce

Against any 5.2.0 instance whose scheduler has registered at least one job, authenticated as a `super_user`:

```jsonc
// 200 — the table is listed in the response
{ "operation": "describe_database", "database": "system" }

// 403 for each of these
{ "operation": "describe_table", "database": "system", "table": "hdb_scheduler_state", "skip_record_count": true }
{ "operation": "search_by_hash", "database": "system", "table": "hdb_scheduler_state", "hash_values": ["leader"], "get_attributes": ["*"] }
{ "operation": "search_by_value", "database": "system", "table": "hdb_scheduler_state", "search_attribute": "id", "search_value": "*" }
{ "operation": "search_by_conditions", "database": "system", "table": "hdb_scheduler_state", "operator": "and", "conditions": [ /* … */ ] }
{ "operation": "sql", "sql": "SELECT * FROM system.hdb_scheduler_state LIMIT 1" }
```

Response body in every failing case:

```json
{
"error": "This operation is not authorized due to role restrictions and/or invalid database items",
"unauthorized_access": [],
"invalid_schema_items": ["Table 'system.hdb_scheduler_state' does not exist"]
}
```

`cluster_status` returns 200 on the same credentials in the same session, so this is not an authentication or connectivity problem.

## Expected

Either the table is readable by a `super_user`, or the refusal says so. `does not exist` for a table the API itself enumerates is misleading in both directions.

## Analysis

`unauthorized_access` is empty and the message is `TABLE_NOT_FOUND`, which identifies the branch: in `utility/operation_authorization.ts`, `hasPermissions()` emits `SCHEMA_NOT_FOUND` at ~line 753 when `userPerms[schema]` is missing, and `TABLE_NOT_FOUND` at ~line 771 when the schema entry exists but the table is absent from `userPerms[schema].tables`.

Since the error is `TABLE_NOT_FOUND`, the expanded role carries a `system` entry with a `.tables` map, and the five tables are simply **not in it**. The eighteen that resolve are present.

This looks like an ordering problem: the per-table permission map is built from the schema as it exists at role-resolution time, whereas these five are created later by `table({ database: 'system', … })` when their component first needs them — for the scheduler, `getStateTable()` is only reached after `startSchedulerEngine()`, itself gated on `getWorkerIndex() === 0`. `describe_database` escapes it because it carries no table in its permission map and enumerates the live registry instead, which is why the table is visible but unaddressable.

Note that `super_user` deliberately does not receive the blanket bypass for system-schema operations (`if (isSuperUser && !isSuSystemOperation) return null`), so there is no path around the per-table lookup for these.

### Ruled out

- **A denylist.** None exists; none of the five is named anywhere outside its own subsystem.
- **`NON_REPLICATING_SYSTEM_TABLES`.** Contains `hdb_temp`, `hdb_job`, `hdb_info` — all accessible — and omits four of the five gated tables.
- **`SYSTEM_TABLE_NAMES`.** 12 entries, but 18 tables are accessible.
- **Role configuration.** `user_info` reports `{ super_user: true }`; `unauthorized_access` is empty in every response.
- **Worker-thread routing.** Deterministic across 16 repeats in both directions.

## Impact

Harper's own scheduler state is unobservable through the operations API. Any tooling that wants to confirm the native scheduler is healthy — leader lease, last run, last status per job — has no supported way to read it remotely, and the error message actively points investigators at the wrong conclusion. It cost us a debugging cycle chasing a supposed renamed-table theory before we traced the authorization path; `hdb_scheduler_state` is current in 5.2.0 and referenced in `resources/scheduler/engine.ts`.

The same applies to `hdb_status` and `hdb_session`, which are natural targets for operational monitoring.

## Suggested directions

1. Register component-created system tables in the permission map when they are created, rather than only at role-resolution time.
2. Failing that, allow `super_user` reads of any table present in the live registry for the `system` database.
3. Independently: distinguish "table not found" from "not permitted" in the response. The current message is wrong whenever the table demonstrably exists.

Issue generated by kAIle (Claude Opus 5)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.