[Performance] Late-game HEAPMNG Total grows far above Committed memory
- Dominant language
- Lua
- Stars
- 264
- Forks
- 260
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 21
Description
## Summary
In long, heavy AI/mod games, FAF's periodic memory log can show **Heap Total** continuing to grow in approximately 31-32 MiB steps while **Heap Committed** remains much lower.
This consumes scarce virtual address space in the 32-bit process and materially increases the chance of a late-game crash.
FAF logs the values in this order:
```text
Heap: Total / Committed
```
Source:
https://github.com/FAForever/fa/blob/develop/lua/system/logger.lua
## Stress-test environment
The documented runs used a severe practical workload rather than a deterministic vanilla reproduction:
- FAF game version 3836 / `fafdevelop`;
- Seton's Clutch FAF v0004;
- two private M28-derived Rush AI brains;
- unit cap 1250;
- a large SIM-mod set;
- long observer-mode sessions.
The exact available logs, mod information, executable hashes, and extracted Heap samples are in the attached evidence archive.
## Observed behavior
| Run | Same-tick queue-length cache | Executable identity | Selected late value |
|---|---:|---|---:|
| Reference high-growth run | Off | Separate reference build | `1.893 GiB / 1.300 GiB` |
| Cache-active counterexample | On | Not recoverable from the log | `1.930 GiB / 1.256 GiB` |
| Lowest-growth documented run | On | Allocator candidate `22cb5373...` | `1.406 GiB / 1.326 GiB` |
In the lowest-growth run, Heap Total reached `1.406 GiB` at session time `00:39:07` and remained at that value through at least `00:48:08`, while Committed remained around `1.30-1.33 GiB`.
The low-growth run therefore used about `0.524 GiB` less Heap Total than the cache-active high-growth run, despite having a slightly higher selected Committed value.
That result is consistent with a large difference in allocator expansion and reuse, rather than merely fewer live objects.
## Expected behavior
When HEAPMNG already has suitable free regions, Heap Total should not continue acquiring additional approximately 31-32 MiB regions while Committed remains substantially lower.
The practical goal is to avoid hundreds of MiB of unnecessary reserved address space and preserve headroom for graphics, mappings, DLLs, stacks, and other users of the 32-bit process.
## Mitigation that produced the lowest observed growth
The best documented result used two coordinated changes:
1. **M28-side pressure reduction**
A private M28-derived build reused only the integer command-queue length for the same unit within the same simulation tick and invalidated it whenever that private M28-derived code changed the unit's commands.
AI-side record:
https://github.com/maudlin27/M28AI/issues/394
2. **HEAPMNG bookkeeping/reuse candidate**
A local executable candidate changed:
- the page-map allocation from `0x3FF000` to `0x400000` bytes;
- the right-neighbour coalescing bound from `0xC0000` pages to `0x100000` pages.
Allocator-side record:
https://github.com/FAForever/FA-Binary-Patches/issues/163
The cache-active counterexample shows that the AI-side cache alone is not sufficient to explain the full reduction. The available runs are also not a completed controlled 2x2 experiment, so the independent contribution of each component remains unisolated.
## Reproduction and validation plan
Use one frozen workload and repeat each cell at least three times:
| Cell | Executable | Same-unit/same-tick queue-length cache |
|---|---|---|
| A | Baseline | Off |
| B | Allocator candidate | Off |
| C | Baseline | On |
| D | Allocator candidate | On |
Hold constant:
- map and map version;
- seed;
- AI count and personality;
- unit cap;
- mod list;
- graphics configuration;
- duration;
- executable SHA-256;
- private-fork source identity.
Record:
- Heap Total and Heap Committed over time;
- process virtual size and private bytes;
- real `GetCommandQueue()` calls;
- cache hits, misses, and invalidations;
- HEAPMNG growth reservations;
- largest reusable free region before growth;
- high-address free-region coalescing events.
## Scope and limitations
- This is a severe stress case, not a deterministic vanilla reproduction.
- The cache alone is not proven to cause the full reduction.
- Neither allocator constant has been isolated independently.
- The mitigation reduces the chance of entering the dangerous high-memory region; it does not make every path above 2 GiB safe.
- Detailed failed experiments and crash diagnostics are retained in the technical archive, not in this game-level issue.
## Related records
- HEAPMNG implementation:
https://github.com/FAForever/FA-Binary-Patches/issues/163
- M28 optimization:
https://github.com/maudlin27/M28AI/issues/394
- Historical `0x0095854F` crash index:
https://github.com/FAForever/fa/issues/1445
## Attachment
The existing evidence archive contains text and CSV material only:
[UPLOAD_1_FA_MAIN_EVIDENCE.zip](https://github.com/user-attachments/files/30387459/UPLOAD_1_FA_MAIN_EVIDENCE.zip)
Contributor guide
Research direction
Start with lua/system/logger.lua to verify how Heap Total and Heap Committed are recorded, then review the attached evidence archive and the linked HEAPMNG and M28 records. Reproduce the four-cell validation plan with a frozen workload and repeated runs; done means the allocator and cache effects are separately measured and any reduction in unnecessary Heap Total growth is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- lua
- Domain
- game-dev, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100