try-runtime-cli Crashes Due to OOM (Out-of-Memory) & Long Execution Time – Resolved with 32GB RAM

Description

We encounter critical resource and performance issues when running try-runtime-cli with our target runtimes, which improve drastically only after upgrading system memory. Key details are as follows:

1. Memory Exhaustion & OOM Termination: With the initial 16GB physical memory configuration, the system rapidly runs out of memory during try-runtime execution, causing severe system lag. Eventually, the try-runtime process is consistently killed by the operating system due to an Out-of-Memory (OOM) error.
2. Extreme Execution Time (16GB RAM Scenario): In the rare cases where the process avoids OOM and completes successfully with 16GB RAM, execution takes approximately 3 hours—far exceeding acceptable workflow limits.
3. Resolution with Increased RAM: After upgrading the system’s physical memory to 32GB (keeping other configurations unchanged), the issue is resolved: try-runtime completes execution quickly (no specific long delays observed) and no OOM crashes occur.

Environment Configuration

Additional Context

This behavior suggests the try-runtime process (especially when handling the ~3.1GB bifrost-polkadot snapshot and Bifrost-specific runtimes) has a higher minimum memory requirement than initially anticipated. While upgrading to 32GB RAM resolves the issue for us, we have questions for further optimization:

1. Is the 32GB RAM requirement expected for try-runtime-cli 0.8.0 when processing large snapshots (e.g., 3.1GB) for Bifrost runtimes?
2. Are there CLI flags, runtime configurations, or snapshot optimization techniques (e.g., pruning) that could reduce the memory footprint, allowing reliable execution with 16GB RAM?
3. Could the high memory usage be linked to runtime-specific logic in bifrost-polkadot/bifrost-paseo (e.g., unoptimized state handling during try-runtime validation) rather than just try-runtime-cli itself?

Steps to Reproduce

1. Reproduce OOM/Long Execution (16GB RAM):
   • Use try-runtime-cli 0.8.0 to run validation against bifrost-polkadot (with 3.1GB snapshot) on a system with 8 cores, 16GB RAM, and 1632GB swap.
   • Monitor memory usage: Observe rapid memory depletion, system lag, and eventual OOM killing. If successful, execution takes ~3 hours.
2. Verify Resolution (32GB RAM):
   • Upgrade the same system’s physical memory to 32GB.
   • Re-run the same try-runtime command: Execution completes quickly with no OOM errors.

No source files, tests, or entry points are named. Start by reproducing try-runtime-cli 0.8.0 with the bifrost-polkadot ~3.1GB snapshot under the documented 16GB and 32GB configurations, then determine whether the memory and runtime differences originate in the CLI or target runtime; done means identifying a reproducible cause or actionable mitigation.