Benchmarking runtimes

Top level issue tracking the improvements/additions needed to track a full-frame workflow runtime.

Current knobs to test are

1. Using CPU vs GPU
2. SLC input stack size
3. Output posting (or equivalently strides)
4. Size of blocks loaded at one time from the input stack
5. For CPU, number of CPUs/number of threads per CPU
6. Algorithm for phase linking (MLE vs EVD)

I've put these in the order of my guess for which will have the biggest effect, but we clearly need to do the tests to see.

Things we need for good testing

• the single-update workflow script (#11)
• recording the threads (#28)
• recording the block size/fixing the max_ram_gb option (#32)
• adding the ability to use EVD instead of MLE (#138 )
• using vmtouch -e on the SLC stack before starting the workflow: https://github.com/hoytech/vmtouch . This will make sure we don't get very fast runs just because the SLC data as cached, as we can't count on that happening for the production runs.

Start by reviewing the single-update workflow script (#11), thread recording (#28), block-size and max_ram_gb work (#32), and EVD support (#138). Then define how runs will vary CPU/GPU, SLC stack size, output posting, block size, CPU threads, and phase-linking algorithm; done means comparable runtime measurements for those variables, with the SLC stack evicted via vmtouch -e.