Towards a systematic benchmarking pipeline

# What metrics should we benchmark on?

## Compressibility
Bytes of the compressed video/bytes of the raw video
One should also consider how easy it is to transform this format from archival to usable (e.g. seek-ability may be an issue)

## Quality
### Unsupervised-methods
1) VMAF (https://en.wikipedia.org/wiki/Video_Multimethod_Assessment_Fusion) this is largely to validate video compression for streaming services. It optimizes human vision, so it is unclear whether we are losing something critical or not.

### Supervised-methods
#### Pose estimation methods
1) Make a labeled dataset from uncompressed video (this will be used as ground truth to extract positions for all parts)
2) Choose a pose-estimation tracking algorithm (or multiple).
3) Choose all the space of encoding parameters to use (e.g. spatially decimate, any codec, correct gamma, etc...)
4) Train one model for each combination of settings to test
5) Compare the performance of all models against held-out ground truth data in 1)

## Online encoding performance
This largely depends on the exact benchmark one is trying to hit as well as the hardware available. Given the current needs of the behavior team, let us aim to be able to:
- 2 simultaneous cameras of 1.3M each at 500Hz.
- 4 cameras of 1.3M at 120hz

## Availability
Concerned with how easy it would be to build a pipeline on top of it. Some of the points I would be worried about include:
- Is it widely available across different platforms and architectures;
- Is it open-source
- Can it be re-distributed?

## Other questions

# Benchmarking as a platform/service

Another way to approach this issue is to build on top of the idea that seeded this repository (i.e. have a turn key solution to collect video data systematically and under the same format).
In short, we should acknowledge that there isn't likely to be a silver bullet when it comes to compressing video and instead create a platform that, given a certain list of requirements (e.g. resolution, and fps) can ingest user-provided data and automatically run through all the benchmarks to provide a comprehensive report of pros/cons of all the supported methods. It is then up to the project to decide what trade-off to take. This solves a few problems:
- Multiple people all over the world can contribute with new benchmarks / annotated videos;
- We no longer need to tailor to a non-fully determined list of specifications;
- We may be able to, from RAW (or other format taken as ground-truth) re-encode the videos automatically without relaying on online acquisition systems. This is not a replacement for the "real-thing" but it may be a good heuristic
- New videos collections, despite having potentially similar specifications of hardware and video formats (e.g. fps, resolution, bit-depth...), are likely to exhibit different spatial and temporal statistics. This search method ensures that an encoding procedure can be optimized for each video
- With enough data, we might even conclude that using different algorithms it is just not worth it and there may indeed be a silver bullet.

This may actually be well-suited for a decentralized platform across institutions.