huggingface / huggingface/evaluate
Loading metrics that are shipped with the evaluate package is slow
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Loading metrics that are shipped with the evaluate package takes way too long to load, up to or more than a second whereas I expect it to be near instant.
Repro:
Run the following in a jupyter notebook:
`import evaluate`
```
%%prun
evaluate.load("accuracy")
```
This outputs the following, suggesting that even for this metric that is available in the package [itself ](https://github.com/huggingface/evaluate/blob/main/metrics/accuracy/accuracy.py)it sets up all kinds of communication with the HF hub:
```
22184 function calls (21809 primitive calls) in 1.351 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
2 0.643 0.322 0.643 0.322 {method 'load_verify_locations' of '_ssl._SSLContext' objects}
2 0.240 0.120 0.240 0.120 {method 'read' of '_ssl._SSLSocket' objects}
2 0.223 0.112 0.223 0.112 {method 'do_handshake' of '_ssl._SSLSocket' objects}
2 0.145 0.073 0.145 0.073 {method 'connect' of '_socket.socket' objects}
2 0.038 0.019 0.038 0.019 {built-in method _socket.getaddrinfo}
80 0.009 0.000 0.009 0.000 {built-in method nt.stat}
45 0.006 0.000 0.006 0.000 {built-in method __new__ of type object at 0x00007FF81939AD50}
```
Contributor guide
Assessment
This issue has not been assessed yet.