Encrypt private ledger data directly into serialised entries
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 876
- Forks
- 260
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 157
Description
## Background
A `perf` profile of the 3,000,000-request `pi_basic` workload attributes approximately 24.06% inclusive CPU to `CommittableTx::serialise()`. Across the full profile, memory copy/set leaves account for 5.06% and allocation/refcount leaves for 16.25%; within transaction serialization alone these categories account for approximately 1.08% and 2.02% of total CPU respectively.
The current private-domain serialization path creates and copies through several buffers:
1. `GenericSerialiseWrapper` builds separate public and private writer buffers.
2. `serialise_domains()` allocates the final ledger entry.
3. It allocates a `serialised_hdr` vector and an `encrypted_private_domain` vector.
4. `KeyAesGcm_OpenSSL::encrypt()` allocates another local ciphertext vector, fills it, and move-assigns it over the caller-provided vector.
5. The GCM header, public domain, and encrypted private domain are copied into the final ledger entry.
This work is separate from reusing initialised AES-GCM contexts (#8168) and should be implemented/measured independently.
## Proposed implementation
- Add a span/caller-owned-output encryption API alongside or beneath the existing vector API.
- Precompute the complete serialised-entry layout and allocate the final ledger entry once.
- Write the entry header, public-domain size, and public-domain bytes directly to their final positions.
- Encrypt the private domain directly into its final ciphertext span in the ledger entry, while continuing to authenticate the public domain as AAD.
- Serialize the fixed-size GCM header/tag directly into the reserved header region once encryption completes.
- Remove the local `ciphertext` allocation in `KeyAesGcm_OpenSSL::encrypt()` when the caller already provides correctly sized output storage.
- Consider moving/taking writer buffers where direct output is not possible, but keep that secondary to the single-allocation final layout.
## Compatibility and correctness coverage
- Preserve the ledger entry format byte-for-byte; this must not require a compatibility-version change.
- Verify public-only, private-only, mixed-domain, and empty-private-domain transactions.
- Verify snapshots and historical-secret serialization/decryption.
- Verify ledger replay and recovery against entries produced before and after the change.
- Ensure encryption failure does not expose or replicate a partially constructed entry.
- Preserve the existing rule that public-domain bytes are GCM additional authenticated data.
- Retain vector-based public APIs where required, implemented via the new output-span primitive if practical.
## Measurement
Add focused crypto/serialization benchmarks for small transaction payloads and compare allocation counts, bytes copied, and throughput. Run an A/B `pi_basic` benchmark/profile independently from #8168 so context-reuse and reduced-copy gains remain attributable.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace CommittableTx::serialise(), GenericSerialiseWrapper, serialise_domains(), and KeyAesGcm_OpenSSL::encrypt() to understand the current allocation and copy flow. Use the pi_basic workload and the listed public-only, private-only, mixed, snapshot, replay, and recovery cases as validation. Done means the ledger format remains byte-for-byte compatible, authenticated public-domain data is preserved, failures do not expose partial entries, and focused benchmarks measure allocation, copying, and throughput changes independently from #8168.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- cryptography, distributed-systems, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100