benchmarks

It can happen that changes to code can result in examples running much slower, especially if answers that used to be cached no longer are, resulting in multiple slow computations of the same thing. Hence it would be good if all examples and tests in packages were timed, and times were logged regularly, to detect things slowing down. The detection should be done automatically by installPackage, and there should be a simple way to log the current running times with a new option to installPackage. The results could be stored in the package's auxiliary directory; we could eliminate the AuxiliaryFiles option to newPackage to make things easier.

It would be important to record running times in a way that takes the speed of the processor into account, perhaps by dividing the time to execute the example by the time to execute some standard benchmark of the underlying hardware. Experiments might be done to find the standard benchmark that correlates best with gb computation times.

Start with the installPackage and newPackage entry points and inspect how package examples, tests, auxiliary files, and the AuxiliaryFiles option are handled. Determine how all examples and tests could be timed, how processor speed should be normalized, and where results would be stored. Done means automatic timing during installation plus an option to log current timings.