Nothing limits how many Python processes run at once

Apollo starts a Python child process for every request and never limits how many run at once. That is fine most of the time, because most of the time there are only a handful. Under enough simultaneous chat traffic it is not fine: the processes pile up, the container runs out of memory and is killed.

This happened in production. The container died silently, with nothing in its own logs, which is what an out-of-memory kill looks like from the inside. It came back, took traffic again, and died again, seven times in one hour. Raising the memory limit stopped it, and that was the right call at the time, but it moves the ceiling rather than removing it. A slightly busier hour finds the new one.

What makes it worse than a slow service is that everyone pays. When the container goes, every request in flight goes with it, and callers get load balancer errors while it restarts rather than anything they can act on. One burst of usage takes out the whole service for everyone else.

The fix is to bound the number of children. Past that bound a request either waits its turn or is turned away with something retryable, and the container stays alive. Which of those two, and what the bound should be, is a real decision rather than an obvious one: queueing keeps requests alive but makes them slow, and rejecting is honest but visible to the user.

Two related notes.

The cancellation added in 3.1.1 helps a little here. A child whose caller has gone away is now killed rather than left running, so memory comes back sooner during exactly this kind of storm. It does not cap anything though.

We also do not know the current capacity. Nobody can currently say how many simultaneous chat calls Apollo supports before it dies, which makes it hard to plan for a busy session. Worth measuring as part of this, since the number is what decides the bound.