OpenHFT / OpenHFT/Zero-Allocation-Hashing
Design a complete post-Unsafe memory-access architecture
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 845
- Forks
- 140
- Avg merge
- 12d 38m
- Merged PRs (30d)
- 2
Description
Context
Issue #67 proposed replacing Unsafe with VarHandle, motivated by a large-file
performance result. The bounded experiment in #121 tested a different and much
narrower design: adapting the existing absolute-long-offset Access contract
to byte-array view VarHandles for XXH3.
That adapter was output-equivalent for the exercised cases but slower throughout
the recorded JDK 11/17/21/25 matrix. It was rejected as a drop-in replacement.
The experiment does not answer how this library should operate after Unsafe
memory-access methods are denied or removed.
Research archive: https://github.com/OpenHFT/Zero-Allocation-Hashing/tree/2339c0db7fdac48b7564e93aeeb732e6bee0af95/research/issue-67
Goal
Design and validate the complete supported-memory-access architecture before
selecting an implementation. Do not assume in advance that one abstraction must
serve every input form.
Required investigation
- Inventory every public hashing entry point and its underlying access path:
primitive arrays,byte[],CharSequence/String, heap and direct
ByteBuffer, raw addresses and customAccessimplementations. - Compare at least:
- VarHandle for heap arrays with
MemorySegmentfor off-heap access; MemorySegmentfor both heap and off-heap access where available; and- specialised algorithm/input paths instead of one generic adapter.
- VarHandle for heap arrays with
- Use each API's natural coordinate model. Do not put a per-read
absolute-longto relative-intconversion into a candidate unless that cost
is deliberately being evaluated. - Preserve the Java 8 artefact. Evaluate a modern-JDK MR-JAR path without making
production contents depend on the JDK that runs Maven. - Do not use reflection in hashing hot paths or treat reflection as a viable
fallback for direct-buffer access. - Keep the JPMS work in #60 separate from the memory-access implementation.
Correctness and compatibility evidence
- Prove all established hash values remain unchanged across all public input
forms and algorithm variants, including tuple/128-bit functions. - Exercise algorithm boundaries, unaligned ranges and inputs of at least 4 KiB,
plus a large streaming/file workload comparable with the original report. - Run the modern path in a forked JVM with
--sun-misc-unsafe-memory-access=deny; it must not initialise or invoke Unsafe
memory-access methods. - Document any API, source, binary, archive-layout or runtime-option changes.
Performance evidence
- Benchmark at least XxHash and XXH3.
- Cover
byte[], heapByteBuffer, directByteBuffer, relevant raw/off-heap
access, aligned and unaligned starts, algorithm threshold sizes and a large
working set. - Test representative JDKs including 11, 17, 21 and 25, retaining Java 8 Unsafe
results as the legacy baseline where meaningful. - Retain exact commands, commit SHA, raw JMH output, CPU/OS/JDK details, JVM
arguments, forks, warm-up/measurement settings and confidence intervals. - Benchmark direct calls; a reflective hot-path fallback is not acceptable.
Completion criteria
- A documented architecture decision covering every supported input form.
- A reproducible correctness and performance comparison of viable candidates.
- A demonstrated modern path that operates under Unsafe denial.
- A deterministic packaging plan retaining Java 8 compatibility.
- An explicit go/no-go decision, including documented limitations and follow-up
work if no candidate is yet suitable.
Follows #67 and the rejected experiment in #121.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the research archive for issue #67 and the public hashing entry points listed in this issue, covering arrays, strings, buffers, raw addresses, and custom Access implementations. Compare the stated VarHandle, MemorySegment, specialised-path, and MR-JAR candidates, then run the required correctness and benchmark workloads. Done means a documented architecture decision, reproducible evidence, Unsafe-denial validation, and a deterministic Java 8-compatible packaging plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100