huggingface / huggingface/evaluate

Too Long Package Import

Open
#387 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Package `evaluate` takes too much time to import while it is not so heavy as TensorFlow or Torch.

```shell
$ time python -c 'import torch; print(torch.ones(3))' >> /dev/null
python -c 'import torch; print(torch.ones(3))' 0,84s user 0,06s system 95% cpu 0,942 total
$ time python -c 'import evaluate'
python -c 'import evaluate' 2,23s user 0,19s system 98% cpu 2,453 total
```

It takes 2.5 seconds against 0.9 seconds for PyTorch! Moreover CLI takes the same time and fails on startup.

```shell
$ time evaluate-cli
Traceback (most recent call last):
File "/usr/bin/evaluate-cli", line 8, in
sys.exit(main())
File "/usr/lib/python3.10/site-packages/evaluate/commands/evaluate_cli.py", line 72, in main
if args["module_type"] not in ["metric", "comparison", "measurement"]:
KeyError: 'module_type'
evaluate-cli 2,31s user 0,17s system 99% cpu 2,490 total
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.