hasura / hasura/graphql-engine
Support very large HASURA_GRAPHQL_EVENTS_HTTP_POOL_SIZE
- Dominant language
- TypeScript
- Stars
- 32.1k
- Forks
- 3k
- PR merge metrics
- PR metrics pending
Description
Users with heavy usage of events and, in particular, where their webhooks take a long time to return need a much larger pool size than we've been considering (e.g. by [Little's Law](https://en.wikipedia.org/wiki/Little%27s_law) a webhook that takes a mean of 30s, where events are generated at 100/s, requires a POOL_SIZE of at least 3000 to keep up; although this pool may be distributed among hasura instances)
There are several issues currently:
- the defaults for our `http-client` won't cache more than 10 TCP connections at the moment; this is probably okay if the only use-case is slow webhooks but is just wasteful/slow
- POOL_SIZE > 1,000 or so begins to behave poorly
On the latter point, we see threads "bunch" resulting in bursty work for webhook and poor latency. In the graphs below we test with POOL_SIZE=10,000, each point is a completed event, **X axis is time processEvent was forked** and the Y axis is **time from forking to completion**

I did a deep dive, tearing out and rewriting lots of code (removing http-client, ekg code, etc. from the equation altogether, changing our blocking mechanism to use a fair MVar instead of STM/retry, etc), ending up staring at eventlogs which seemed [sort of suspicious](https://gitlab.haskell.org/ghc/ghc/-/issues/18224).
I noticed that `+RTS -qm` which disables thread rebalancing seems to fix things (and all cores seem to have work to do):
**EDIT**: actually no (see below)

## Remaining tasks
- [x] Test above again with my local node test webhook (above is directing towards a non-existent host); `-qm` still working?
- [x] monitor open file descriptors (are we getting up to 10,000?)
- [x] is the delay in the time from forking to starting thread, or during processing itself? Difficult to tell from threadscope
- [ ] consider opening a GHC issue, with self-contained repro; coordinate with Matt on this
- (**EDIT** N/A) Does `-qm` hurt performance of regular graphql query workloads? (i.e. can we use it?)
- [ ] Does hasura/ghc RTS use `select()`/`pselect()` anywhere? (i.e. [can we even run with higher FD ulimit?](http://0pointer.net/blog/file-descriptor-limits.html))
- **EDIT** I see the RTS uses `select()` on windows, but that's not a blocker
- (**EDIT** N/A) Continue stress testing with `-qm` to find a new good upper bound on POOL_SIZE
- Try some other techniques for more consistent and better throughput (see below: **Factor out the forking...**)
### Changes to make (assuming we can run with > 1024 FD limit)
- [ ] Clamp POOL_SIZE to a reasonable upper bound, warning if requested is too high; this needs to take into account the reasonable upper bound we determined above, as well as the hard RLIMIT at runtime
- [ ] Document POOL_SIZE and BATCH_SIZE better, mention Little's Law
- [ ] Use LD_PRELOAD trick to make `select()` and `pselect()` segfault in our test suite to keep them out
- [ ] On startup bump our `RLIMIT_NOFILE` soft limit to something like 2*POOL_SIZE + 1000
- [ ] set `HTTP.managerConnCount` and `managerIdleConnectionCount` to POOL_SIZE
- Factor out the forking pattern in `processEventQueue.go` into a library function:
- [x] remove the call to `Async.link` which forks a new process (wasteful), and also isn't really what we want; instead just log the unexpected exception (**EDIT** I don't observe much difference, but this will go away regardless)
- [ ] measure using MVar again to avoid the wake-all behavior
- [x] measure yielding manually again after forking N worker threads (or does RTS already consider this when timeslicing?) (**EDIT** even yielding after every two forks doesn't change bunchiness, hmmm...)
- [ ] Test above with random delay in webhook both enabled and disabled
Contributor guide
Assessment
This issue has not been assessed yet.