Create performance tests / benchmarking

In order to keep track of the performance of the library, we should implement performance tests.

It would also allow us to properly assess whether a change actually improves the performance or not.

A first approach we could choose would be to write tests, with proper fixtures to make the test runs independant, using the [pytest-benchmark](https://github.com/ionelmc/pytest-benchmark) extension. This would make running the tests a slower but would allow us to easily benchmarks certain parts of the code without writing too much extra code. Of course we should use pytest markers to prevent running such tests by default.

A second approach would be to use [airspeed-velocity](https://asv.readthedocs.io/en/latest/index.html), a tool for benchmarking Python packages over their lifetime. Runtime, memory consumption and even custom-computed values may be tracked. This would require us to write a bit of code and to ideally keep it in a separate repository, but it would give allow us to track the changes overtime and to track multiple things more easily.

For example, [Dask](https://github.com/dask/dask) is using it and storing the code and results in [this repository](https://github.com/dask/dask-benchmarks).