Sled causing some issues with deterministic benchmarks

Found an issue with sled & the callgrind benchmarks yesterday. It turns out sled has a weird interaction with callgrind that makes the module benchmarks ~7x slower. However, this is ONLY true when running under callgrind. It also doesn't affect benchmarks that don't load a module.

Specifically, sled creates some background worker thread, and throws work on them every so often on a timer. This works fine normally. However, under callgrind, code executes much more slowly, so this timer goes off proportionally much more often, and the background threads can't keep up under the weight of all the instrumentation. The problematic code is here: https://github.com/spacejam/sled/blob/main/src/threadpool.rs

This means an individual benchmark may take 3 minutes, rather than 20 seconds. However, the benchmark results are unchanged, thanks to the magic of callgrind filtering.

Possible solutions:

• Fix sled
  • Add a feature flag to stub sled out with a mock. (I've already implemented the mock, it works. We don't need to mock the actual storage engine used to store user tables since that isn't sled; just all the sled::Dbs that are lying around in ControlDb, Scheduler, etc.)
  • Vendor and patch sled to avoid this issue.
  • Attempt to get a patch merged with upstream sled.
• Don't fix sled
  • I can just minimize the number of benchmarks that hit this issue. It only affects benchmarks that load a compiled module, not the raw database benchmarks.
• Try another datastore???
  • Specifically, everywhere that uses a sled::Db would need to be patched...