1B and 100B Account Performance Testing

## Overview

Validate that the Hiero Consensus Node can sustain 10K TPS with 1 billion accounts (~4-5B total entities) on Mainnet-class hardware, and identify scaling limits at 100 billion accounts on enhanced hardware. This Epic covers the performance testing automation, infrastructure provisioning, and result analysis needed to achieve and validate these targets.

## Background

### Current State

The CITR (Continuous Integration Test and Release) framework supports account-scale longevity testing via the Single Day Longevity Test (SDLT) pipeline, which runs sustained load tests at scales from 100K to 1B accounts. However:

- **Current CITR test infrastructure** is undersized relative to mainnet: 20 vCPU / 32 GB RAM per CN vs mainnet's 48 vCPU / 256 GB RAM (`n2d-custom-48-262144`)
- **Java heap** in CITR is limited to 30 GB (`-Xmx30g`) vs mainnet's 150 GB (`-Xmx150g`, `-Xms32g`)
- **1B account tests** have been run experimentally (see #22667) but are not yet stabilized in the automated pipeline
- **100B accounts** have not been tested; the current `MerkleDbConfig.maxNumOfKeys` default of 8 billion supports ~4-5B entities but would need significant increases for 100B accounts

### MerkleDB Configuration

Key parameters from [`MerkleDbConfig.java`](https://github.com/hiero-ledger/hiero-consensus-node/blob/main/platform-sdk/swirlds-merkledb/src/main/java/com/swirlds/merkledb/config/MerkleDbConfig.java):

| Parameter | Default | Relevance |
|-----------|---------|-----------|
| `initialCapacity` | 1,000,000,000 (1B) | Initial hash map capacity — sized for 1B entities |
| `maxNumOfKeys` | 8,000,000,000 (8B) | Maximum key count — supports ~4-5B entities (accounts + token associations) |
| `compactionThreads` | 3 | Compaction parallelism — may need tuning at scale |
| `maxCompactionLevel` | 5 | Compaction depth — affects disk I/O patterns |
| `goodAverageBucketEntryCount` | 32 | HDHM bucket sizing — affects memory vs disk tradeoff |
| `useDiskIndices` | false | When true, moves bucket indices to disk to reduce off-heap memory |

The [`HalfDiskHashMap`](https://github.com/hiero-ledger/hiero-consensus-node/blob/main/platform-sdk/swirlds-merkledb/src/main/java/com/swirlds/merkledb/files/hashmap/HalfDiskHashMap.java) keeps bucket indices in RAM with bucket data on disk. At 100B entities, the bucket index alone would require significant memory, potentially necessitating `useDiskIndices=true`.

### Test Infrastructure

| Component | Current (CITR) | Mainnet | Target for This Epic |
|-----------|---------------|---------|---------------------|
| vCPUs per CN | 20 | 48 | 48 |
| RAM per CN | 32 GB | 256 GB | 256 GB |
| Java heap (`-Xmx`) | 30 GB | 150 GB | 150 GB |
| Java min heap (`-Xms`) | — | 32 GB | 32 GB |
| NLG heap | 30 GB | N/A | 30 GB+ |
| Concurrent clients | 32 | — | 32+ |

### CITR Test Types

| Abbreviation | Name | Purpose |
|-------------|------|---------|
| **SDLT** | Single Day Longevity Test | Sustained 5.5-hour load test at 100M–1B accounts |
| **SDPT** | Single Day Performance Test | Ad-hoc performance validation |
| **MQPT** | Merge Queue Performance Test | CI gating — TPS regression detection |
| **MDLT** | Multi-Day Longevity Test | Extended duration stability testing |

Relevant workflow files:
- [`zxc-single-day-longevity-test.yaml`](https://github.com/hiero-ledger/hiero-consensus-node/blob/main/.github/workflows/zxc-single-day-longevity-test.yaml)
- [`zxc-execute-performance-test.yaml`](https://github.com/hiero-ledger/hiero-consensus-node/blob/main/.github/workflows/zxc-execute-performance-test.yaml)
- [`zxc-merge-queue-performance-test.yaml`](https://github.com/hiero-ledger/hiero-consensus-node/blob/main/.github/workflows/zxc-merge-queue-performance-test.yaml)

## Scope of Work

### Phase 1: 1B Account Validation (Primary)

**1.1 Infrastructure Provisioning**
- Provision mainnet-class test nodes (48 vCPU, 256 GB RAM, `-Xmx150g`) in the CITR performance cluster
- Configure MerkleDB for 1B accounts: `initialCapacity=1000000000`, verify `maxNumOfKeys=8000000000` is sufficient
- Ensure NLG (Network Load Generator) can sustain load generation at 1B account scale with 32+ concurrent clients

**1.2 SDLT Stabilization at 1B**
- Stabilize the 1B account SDLT test run (currently experimental, 330-minute / 5.5-hour duration)
- Automate result collection and reporting to GCS (`gs://performance-engineering-reports/`)
- Integrate with Grafana dashboards at the performance monitoring endpoint
- Validate MerkleDB compaction V3 behavior at 1B scale (relates to #24125)

**1.3 TPS Validation**
- Achieve and sustain 10K TPS with 1B accounts on mainnet-class hardware
- Run the following load test suites at 1B scale:
- `CryptoTransferLoadTest` (baseline: ~20K TPS at smaller scales)
- `NftTransferLoadTest` (baseline: ~12K TPS)
- `HCSLoadTest` (baseline: ~20K TPS)
- `TokenTransferLoadTest`
- `SmartContractLoadTest` (baseline: ~8K TPS)
- Measure and report: TPS, p50/p99 latency, memory usage (heap + off-heap), disk I/O, state size growth, compaction duration

**1.4 Longevity Validation**
- Run multi-day longevity tests (MDLT) at 1B accounts to validate stability over extended periods
- Monitor for memory leaks, compaction degradation, and state growth anomalies
- Validate reconnect behavior at 1B scale (relates to #10812)

### Phase 2: 100B Account Feasibility (Stretch)

**2.1 Configuration Analysis**
- Determine required `maxNumOfKeys` for 100B accounts (~125B+ entity slots including token associations)
- Estimate memory requirements for HDHM bucket indices at 100B scale
- Evaluate `useDiskIndices=true` for reducing off-heap memory at the cost of disk I/O
- Identify hardware requirements (enhanced mainnet-class: additional RAM, NVMe storage)

**2.2 Infrastructure Extension**
- Extend SDLT/NLG workflows to support 100B account parameter values
- Provision enhanced hardware for 100B testing (likely requires dedicated bare-metal or large cloud instances)
- Implement incremental state loading (loading 100B accounts takes significantly longer than 1B)

**2.3 Initial Testing**
- Run exploratory performance tests at 10B accounts as intermediate milestone
- Identify and document scaling bottlenecks (hashing time, compaction, disk I/O, memory)
- Provide recommendations for code changes needed to support 100B (feed back to Entity Scalability program #51)

## Success Criteria

### 1B Account (Primary)
- [ ] Sustained 10K TPS with 1B accounts on mainnet-class hardware (48 vCPU, 256 GB RAM)
- [ ] SDLT passes consistently (5.5 hours, no OOM, no compaction failures)
- [ ] MDLT passes (multi-day stability)
- [ ] All five load test suites complete successfully at 1B scale
- [ ] Results automatically reported to Grafana and GCS

### 100B Account (Stretch)
- [ ] Hardware and configuration requirements documented
- [ ] 10B account intermediate test completed
- [ ] Scaling bottlenecks identified and documented
- [ ] Recommendations provided to Entity Scalability program

## Related Issues

- #24125 — 1B Longevity for MerkleDB compaction V3 + JDK25
- #22667 — 1B entities experiment (closed)
- #10812 — Scalability - Reconnect Improvements
- #22674 — HashSphere performance automation for SRE

## References

- [Entity Scalability Program (A8)](https://github.com/hiero-ledger/roadmap/issues/51) — owns code changes for scalability
- [Automation Program (A11)](https://github.com/hiero-ledger/roadmap/issues/49) — owns testing automation
- [Validate 1B Accounts at 10K TPS](https://github.com/hiero-ledger/roadmap/issues/58) — roadmap validation goal