OpenHFT / OpenHFT/Zero-Allocation-Hashing

Design a complete post-Unsafe memory-access architecture

Open
#124 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
845
Forks
140
Avg merge
12d 38m
Merged PRs (30d)
2

Description

Context

Issue #67 proposed replacing Unsafe with VarHandle, motivated by a large-file
performance result. The bounded experiment in #121 tested a different and much
narrower design: adapting the existing absolute-long-offset Access contract
to byte-array view VarHandles for XXH3.

That adapter was output-equivalent for the exercised cases but slower throughout
the recorded JDK 11/17/21/25 matrix. It was rejected as a drop-in replacement.
The experiment does not answer how this library should operate after Unsafe
memory-access methods are denied or removed.

Research archive: https://github.com/OpenHFT/Zero-Allocation-Hashing/tree/2339c0db7fdac48b7564e93aeeb732e6bee0af95/research/issue-67

Goal

Design and validate the complete supported-memory-access architecture before
selecting an implementation. Do not assume in advance that one abstraction must
serve every input form.

Required investigation

  • Inventory every public hashing entry point and its underlying access path:
    primitive arrays, byte[], CharSequence/String, heap and direct
    ByteBuffer, raw addresses and custom Access implementations.
  • Compare at least:
    • VarHandle for heap arrays with MemorySegment for off-heap access;
    • MemorySegment for both heap and off-heap access where available; and
    • specialised algorithm/input paths instead of one generic adapter.
  • Use each API's natural coordinate model. Do not put a per-read
    absolute-long to relative-int conversion into a candidate unless that cost
    is deliberately being evaluated.
  • Preserve the Java 8 artefact. Evaluate a modern-JDK MR-JAR path without making
    production contents depend on the JDK that runs Maven.
  • Do not use reflection in hashing hot paths or treat reflection as a viable
    fallback for direct-buffer access.
  • Keep the JPMS work in #60 separate from the memory-access implementation.

Correctness and compatibility evidence

  • Prove all established hash values remain unchanged across all public input
    forms and algorithm variants, including tuple/128-bit functions.
  • Exercise algorithm boundaries, unaligned ranges and inputs of at least 4 KiB,
    plus a large streaming/file workload comparable with the original report.
  • Run the modern path in a forked JVM with
    --sun-misc-unsafe-memory-access=deny; it must not initialise or invoke Unsafe
    memory-access methods.
  • Document any API, source, binary, archive-layout or runtime-option changes.

Performance evidence

  • Benchmark at least XxHash and XXH3.
  • Cover byte[], heap ByteBuffer, direct ByteBuffer, relevant raw/off-heap
    access, aligned and unaligned starts, algorithm threshold sizes and a large
    working set.
  • Test representative JDKs including 11, 17, 21 and 25, retaining Java 8 Unsafe
    results as the legacy baseline where meaningful.
  • Retain exact commands, commit SHA, raw JMH output, CPU/OS/JDK details, JVM
    arguments, forks, warm-up/measurement settings and confidence intervals.
  • Benchmark direct calls; a reflective hot-path fallback is not acceptable.

Completion criteria

  • A documented architecture decision covering every supported input form.
  • A reproducible correctness and performance comparison of viable candidates.
  • A demonstrated modern path that operates under Unsafe denial.
  • A deterministic packaging plan retaining Java 8 compatibility.
  • An explicit go/no-go decision, including documented limitations and follow-up
    work if no candidate is yet suitable.

Follows #67 and the rejected experiment in #121.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the research archive for issue #67 and the public hashing entry points listed in this issue, covering arrays, strings, buffers, raw addresses, and custom Access implementations. Compare the stated VarHandle, MemorySegment, specialised-path, and MR-JAR candidates, then run the required correctness and benchmark workloads. Done means a documented architecture decision, reproducible evidence, Unsafe-denial validation, and a deterministic Java 8-compatible packaging plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.