huggingface / huggingface/evaluate

Loading metrics that are shipped with the evaluate package is slow

Open
#511 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Loading metrics that are shipped with the evaluate package takes way too long to load, up to or more than a second whereas I expect it to be near instant.

Repro:
Run the following in a jupyter notebook:
`import evaluate`

```
%%prun
evaluate.load("accuracy")
```

This outputs the following, suggesting that even for this metric that is available in the package [itself ](https://github.com/huggingface/evaluate/blob/main/metrics/accuracy/accuracy.py)it sets up all kinds of communication with the HF hub:
```
22184 function calls (21809 primitive calls) in 1.351 seconds

Ordered by: internal time

ncalls tottime percall cumtime percall filename:lineno(function)
2 0.643 0.322 0.643 0.322 {method 'load_verify_locations' of '_ssl._SSLContext' objects}
2 0.240 0.120 0.240 0.120 {method 'read' of '_ssl._SSLSocket' objects}
2 0.223 0.112 0.223 0.112 {method 'do_handshake' of '_ssl._SSLSocket' objects}
2 0.145 0.073 0.145 0.073 {method 'connect' of '_socket.socket' objects}
2 0.038 0.019 0.038 0.019 {built-in method _socket.getaddrinfo}
80 0.009 0.000 0.009 0.000 {built-in method nt.stat}
45 0.006 0.000 0.006 0.000 {built-in method __new__ of type object at 0x00007FF81939AD50}

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.