HarperFast / HarperFast/rocksdb-js
Explore RocksDB Merge operator for conflict-free commutative increments (hot-counter workloads)
- Dominant language
- C++
- Stars
- 21
- Forks
- 2
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 36
Description
## Use case
High-speed, low-cardinality counter updates — e.g. token-quota accounting where many concurrent requests increment a property on the same record. With optimistic transactions, every concurrent writer to a hot key conflicts and retries; even with coordinated retry (verification-table parking), throughput on a single hot record is fundamentally serialized read-modify-write.
## Idea
RocksDB's native **Merge operator** is built for exactly this: writers append merge operands (deltas) instead of reading + rewriting the value, so concurrent increments are conflict-free by construction. The merge function applies the operand chain at read/compaction time.
For rocksdb-js this would mean:
- Exposing a `merge(key, operand)` write on stores/transactions.
- A msgpackr-aware merge operator in native code that can apply field-level numeric deltas to an encoded record — or, more simply, a dedicated column family for counters with plain numeric encoding and a stock int64-add operator.
## Costs / open questions (why this is exploratory)
- Read path must apply the merge-operand chain (get latency grows with unmerged operands until compaction).
- Harper's replication and audit log currently assume value-images produced at commit time; delta operands would need first-class support through the transaction log — although op-based increments would actually replicate *commutatively* across nodes, which is very attractive for multi-node quota semantics.
- Conflict semantics between merge operands and regular transactional writes to the same key need careful definition.
## Relation to the shallow alternative
A JS-side intent-based increment coalescer (filed in the harper repo) gets most of the single-node win with a fraction of the effort and no storage-format implications. This issue tracks the deeper storage-engine option, gated on the shallow tier proving insufficient (e.g. multi-node commutative replication becoming a requirement).
---
_Filed by KrAIs (Claude Opus 4.8) on behalf of @kriszyp, from the commit-batching feasibility research (2026-07-08)._
Contributor guide
Research direction
The issue names no files or tests; begin by tracing the RocksDB transaction, replication, and audit-log paths, then compare them with the JS-side intent-based increment coalescer mentioned in the issue. Done means establishing whether merge operands can be supported without unresolved conflicts in value images, replication, or regular writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, node.js
- Domain
- backend, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100