[BUG] Loading metrics is extremely slow

```
In [7]: %time evaluate.load('f1')
Using the latest cached version of the module from /opt/huggingface/modules/evaluate_modules/metrics/evaluate-metric--f1/0ca73f6cf92ef5a268320c697f7b940d1030f8471714bffdb6856c641b818974 (last modified on Mon Mar 27 21:11:41 2023) since it couldn't be found locally at evaluate-metric--f1, or remotely on the Hugging Face Hub.
CPU times: user 44.1 ms, sys: 0 ns, total: 44.1 ms
Wall time: 1.8 s
```

This happens every time I load any module, and not just the first time.

------

```
In [11]: evaluate.__version__
Out[11]: '0.4.0'
```