HarperFast / HarperFast/harper

The table-size analytics metric is never emitted on RocksDB

Open
#2,249 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

`METRIC.TABLE_SIZE` (`table-size`) has exactly one emitter, and it sits on the **non-RocksDB** branch of `storeDBSizeMetrics`. Since RocksDB is the default (and LMDB is deprecated), the metric is absent on effectively every current deployment — while being documented without qualification in the analytics reference.

## Mechanism

`resources/analytics/write.ts`:

```ts
if (firstTable.primaryStore instanceof RocksDatabase) {
// ... database-size from SST totals
storeMetric(analyticsTable, metric);
} else {
const dbUsedSize = storeTableSizeMetrics(analyticsTable, db, tables); // <-- only caller
// ... database-size with used/free/audit
}
```

`storeTableSizeMetrics` is the only place that writes `METRIC.TABLE_SIZE`, and line 391 is its only call site. So on RocksDB:

- no `table-size` rows are written at all
- `database-size` is written without `used` / `free` / `audit`

The per-table RocksDB path (`storeRocksDBStatsMetrics`) does emit a per-table row every cycle, but as `rocksdb-stats`, carrying only `numRunningCompactions` / `compactionPending` — no size.

`Table.getSize()` already handles both engines (`rocksdb.estimate-live-data-size` for RocksDB), so the value is available; nothing consumes it on this path.

## Impact

- Any dashboard or alert built on `table-size` is silently empty on a RocksDB install. Absence reads as "no data yet", not "this metric does not exist here".
- [Analytics reference — Resource Usage Metrics](https://docs.harperdb.io/docs/reference/analytics/overview) lists `table-size` with `database`, `table` dimensions and no engine qualifier.
- It also blocks putting anything else per-table on that record. Found while adding record-structure dictionary counts for #2220 — the fields would have been written into a row that is never emitted, so that hunk was dropped.

## Open question before fixing

Emitting a `table-size` row per table per aggregation cycle (60s default) is real cardinality — ~525k rows/table/year in `hdb_analytics`. Worth deciding whether the RocksDB emission should be:

1. per aggregation cycle, matching the LMDB branch, or
2. gated on `analytics.storageInterval` (default every 10th cycle), matching how `storage-volume` is sampled,

before wiring it. Option 2 seems better proportioned to how slowly the value moves; the docs should state whichever is chosen.

Alternatively, if `table-size` is considered obsolete on RocksDB, it should be removed from the reference rather than left documented and dead.

Contributor guide

Open the contributing guide

Research direction

Read resources/analytics/write.ts at storeDBSizeMetrics, comparing the RocksDB and non-RocksDB branches, then inspect Table.getSize(). Decide whether table-size should be emitted per cycle, sampled, or removed; done means the chosen behavior is implemented and the analytics reference matches it.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
analytics, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.