try-runtime-cli Crashes Due to OOM (Out-of-Memory) & Long Execution Time – Resolved with 32GB RAM

Description

We encounter critical resource and performance issues when running try-runtime-cli with our target runtimes, which improve drastically only after upgrading system memory. Key details are as follows:

1. Memory Exhaustion & OOM Termination: With the initial 16GB physical memory configuration, the system rapidly runs out of memory during try-runtime execution, causing severe system lag. Eventually, the try-runtime process is consistently killed by the operating system due to an Out-of-Memory (OOM) error.
2. Extreme Execution Time (16GB RAM Scenario): In the rare cases where the process avoids OOM and completes successfully with 16GB RAM, execution takes approximately 3 hours—far exceeding acceptable workflow limits.
3. Resolution with Increased RAM: After upgrading the system’s physical memory to 32GB (keeping other configurations unchanged), the issue is resolved: try-runtime completes execution quickly (no specific long delays observed) and no OOM crashes occur.

Environment Configuration

Additional Context

This behavior suggests the try-runtime process (especially when handling the ~3.1GB bifrost-polkadot snapshot and Bifrost-specific runtimes) has a higher minimum memory requirement than initially anticipated. While upgrading to 32GB RAM resolves the issue for us, we have questions for further optimization:

1. Is the 32GB RAM requirement expected for try-runtime-cli 0.8.0 when processing large snapshots (e.g., 3.1GB) for Bifrost runtimes?
2. Are there CLI flags, runtime configurations, or snapshot optimization techniques (e.g., pruning) that could reduce the memory footprint, allowing reliable execution with 16GB RAM?
3. Could the high memory usage be linked to runtime-specific logic in bifrost-polkadot/bifrost-paseo (e.g., unoptimized state handling during try-runtime validation) rather than just try-runtime-cli itself?

Steps to Reproduce

1. Reproduce OOM/Long Execution (16GB RAM):
   • Use try-runtime-cli 0.8.0 to run validation against bifrost-polkadot (with 3.1GB snapshot) on a system with 8 cores, 16GB RAM, and 1632GB swap.
   • Monitor memory usage: Observe rapid memory depletion, system lag, and eventual OOM killing. If successful, execution takes ~3 hours.
2. Verify Resolution (32GB RAM):
   • Upgrade the same system’s physical memory to 32GB.
   • Re-run the same try-runtime command: Execution completes quickly with no OOM errors.

Es werden keine Quelldateien, Tests oder Einstiegspunkte genannt. Beginne damit, try-runtime-cli 0.8.0 mit dem ~3.1GB großen bifrost-polkadot-Snapshot unter den dokumentierten Konfigurationen mit 16GB und 32GB zu reproduzieren, und ermittle dann, ob die Unterschiede beim Speicherverbrauch und bei der Runtime aus der CLI oder der Ziel-Runtime stammen; abgeschlossen ist die Aufgabe, wenn eine reproduzierbare Ursache oder eine umsetzbare Abhilfe identifiziert wurde.