Re-scoring previous test runs causes confusion

Recent changes to motion-path and URL tests caused a noticeable overall change in the Firefox score. This is fine; those tests changes were agreed and the score change was predictable. However, what caused some problems was that people saw an overall score of X on one day, and then on the next day saw that not only had the score dropped to < X, but that the graph suggested that the score had never been as high as X in the first place. That caused a lot of confusion.

This happens because we try to rescore previous runs as if we had the current test set (zero filling results for tests that weren't in the previous runs). That's reasonable; it means drops in the graph usually (but not always e.g. in the case that existing tests are edited to have different pass conditions) correspond to actual browser regression. But there are a couple of problems:

• There's a lack of documentation explaining exactly what the system is. To actually understand what's going on in detail the best source is the scoring code itself (which is well commented!), but it's unreasonable to expect most people to find that.

• The fact that old scores are silently changed makes it very difficult to quote a specific score. Consider a press article that says "at time of writing browser B had score Y". Then someone reading the article some time later tries to verify that, and finds a graph that shows B never getting a score of Y. In that situation the reader would likely conclude that the article author had made an error, rather than digging in to the discrepancy.

I don't think the rescoring system is necessarily bad, but I do think we need to do more to make it clear what's going on. In particular the following seems like it would help:

• Clearly document that we re-score previous runs using the current set of tests, backfilling zero where the test doesn't exist, and explain the set of tradeoffs that led to this system.
• Generate, and publish, the actual measured results for each run, with the test set at the time of the run. Functionally the backend for this would just be a new CSV file that we append the latest results to every time a new aligned run is processed. On the frontend, having some way to switch the graphs to show either the re-scored results, or the historic point-in-time results would make it much more transparent what's going on.

Start by reading the well-commented scoring code to understand how previous runs are rescored and how missing tests are zero-filled. Then trace the backend's aligned-run processing and the frontend graph data flow; done means the rescoring tradeoffs are documented and users can compare current rescored results with published point-in-time results.