HarperFast / HarperFast/rocksdb-js

Explore RocksDB Merge operator for conflict-free commutative increments (hot-counter workloads)

Open
#692 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
21
Forks
2
Avg merge
2d 9h
Merged PRs (30d)
36

Description

## Use case

High-speed, low-cardinality counter updates — e.g. token-quota accounting where many concurrent requests increment a property on the same record. With optimistic transactions, every concurrent writer to a hot key conflicts and retries; even with coordinated retry (verification-table parking), throughput on a single hot record is fundamentally serialized read-modify-write.

## Idea

RocksDB's native **Merge operator** is built for exactly this: writers append merge operands (deltas) instead of reading + rewriting the value, so concurrent increments are conflict-free by construction. The merge function applies the operand chain at read/compaction time.

For rocksdb-js this would mean:

- Exposing a `merge(key, operand)` write on stores/transactions.
- A msgpackr-aware merge operator in native code that can apply field-level numeric deltas to an encoded record — or, more simply, a dedicated column family for counters with plain numeric encoding and a stock int64-add operator.

## Costs / open questions (why this is exploratory)

- Read path must apply the merge-operand chain (get latency grows with unmerged operands until compaction).
- Harper's replication and audit log currently assume value-images produced at commit time; delta operands would need first-class support through the transaction log — although op-based increments would actually replicate *commutatively* across nodes, which is very attractive for multi-node quota semantics.
- Conflict semantics between merge operands and regular transactional writes to the same key need careful definition.

## Relation to the shallow alternative

A JS-side intent-based increment coalescer (filed in the harper repo) gets most of the single-node win with a fraction of the effort and no storage-format implications. This issue tracks the deeper storage-engine option, gated on the shallow tier proving insufficient (e.g. multi-node commutative replication becoming a requirement).

---
_Filed by KrAIs (Claude Opus 4.8) on behalf of @kriszyp, from the commit-batching feasibility research (2026-07-08)._

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests; begin by tracing the RocksDB transaction, replication, and audit-log paths, then compare them with the JS-side intent-based increment coalescer mentioned in the issue. Done means establishing whether merge operands can be supported without unresolved conflicts in value images, replication, or regular writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, node.js
Domain
backend, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.