Compiling Vulkan shaders with less than 50% of free memory leads to OOM situation and system freeze

Your system information

• Steam client version (build number or date): 1689034492
• Distribution (e.g. Ubuntu): Ubuntu
• Opted into Steam client beta?: No
• Have you checked for system updates?: Yes
• Steam Logs: steam-logs.tar.gz
• GPU: AMD Radeon RX 5700 XT
• CPU: AMD Ryzen 7 1700X (8 cores, 16 threads)

Please describe your issue in as much detail as possible:

My system does not have swap configured, which means once the RAM is all occupied the system can't simply use swap to get out of an OOM situation.

What I expect to happen: Steam detects how much memory is available and spawns fossilize_replay processes accordingly so as not to get into an OOM situation, instead of spawning a process for each logical CPU.

What actually happens: Steam spawns the maximum number of fossilize_replay processes it thinks the system supports (in my case, 16 processes). As each process takes up ~512 MB of RAM, this means the system needs at least 8 GB of free RAM. Whenever this is not the case, e.g. because you have other apps open, the system freezes and stops responding. I have tried leaving the system in this state for up to 15 minutes, but the only way to get out of this freeze is to perform a reset (ctrl+alt+printscreen+REISUB).

I have experienced this several times and it is consistently reproduceable.

Steps for reproducing this issue:

1. On a system with 16 GB of RAM and a 16-threads CPU, as well as no swap configured, open applications until you have more than 8 GB occupied.
2. Start Vulkan shaders compilation.

Start by locating fossilize_replay, the process involved in Vulkan shader compilation, and reproduce the issue with limited free memory and no swap as described. Investigate how the process count is selected; done means compilation no longer spawns enough workers to exhaust available memory and freeze the system.