Larges pushes fail, because they take too long

We recently had a push of almost 2GB (surprisingly it only took about 3 min for the client to upload it).
Here's an overview of when things broken:

It failed on the last chunk, because that's when the server actually starts doing the heavy lifting: trying to apply the commit.

And here's what I think/know happened:

• Chorus sends the last chunk
  • The resumable server detects that it's the last chunk
  • It calls unbundle
  • Which calls hg incoming, which takes ~3.5m (it gets logged to /var/cache/hgresume/<transaction-ID>.bundle.incoming.async_run)
• Because the request takes so long
  • Chorus cancels the push request after 30s (so Lexbox returns a 499)
  • It's also fairly likely that PHP kills the script, because it takes longer than 30s and/or, because it was cancelled.
  • It's also fairly likely that Cloudflare/the web server tear down the script process 🤷
• So, presumably the PHP script gets torn down while it's waiting for the hg incoming command to finish (which does finish, because it's in its own process)
• Because the PHP script gets torn down, no work actually happens: it never gets to running the command hg unbundle and creating a lockfile for that command (A lockfile is created for hg incoming, but that's a seperate file)
• Because the lockfile doesn't get created, when Chorus retries the push-bundle, the server throws an Exception

Do we want to allow big pushes like this? I think so! So how:

• It's fine if Chorus times out as long as the job actually happens and we returns a more meaningful response. The code tries to return a 200, but fails, because the lockfile it's expecting doesn't exist. The exception is good, because a missing lockfile means nothing is happening.

So we either need to:

1. Move more stuff into an external command that doesn't get torn down 🙁
2. Prevent the PHP script from getting torn down (e.g. move large pushes to a Lexbox Job and make sure we turn off everything that might kill a long PHP script)
3. Make the retries smarter and have them pick up where the last one died

I think 3 sounds like the best bet. Something like:

• Replace the exception-throwing isComplete check, with something that anticipates this senario:
  • If there's no lock file retry the unbundle
  • In the unbundle, detect if hg incoming already ran and if so:
    • Do the necessary validation
    • Then start hg unbundle

Start with HgResumeApi.php, especially the unbundle flow and isComplete check, then read HgRunner.php and AsyncRunner.php around incoming, unbundle, and lockfile handling. Review the Chorus timeout reference to understand the retry path. Done means a large push can retry after the request times out without an exception and the unbundle work completes or returns a meaningful response.