huggingface / huggingface/evaluate

CI parity test is broken due to FileNotFoundError

Open
#632 0 comments 0 reactions 1 assignee Claimed by @albertvillanova View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

CI parity test is broken due to FileNotFoundError: https://github.com/huggingface/evaluate/actions/runs/10827111652/job/30039577026?pr=631
```
FAILED tests/test_trainer_evaluator_parity.py::TestEvaluatorTrainerParity::test_token_classification_parity - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpg8va97zxevaluator_trainer_parity_test/tokenclassification_conll2003_transformers/eval_results.json'
```
```
_________ TestEvaluatorTrainerParity.test_token_classification_parity __________
[gw0] linux -- Python 3.8.18 /opt/hostedtoolcache/Python/3.8.18/x64/bin/python

self =

def test_token_classification_parity(self):
model_name = "hf-internal-testing/tiny-bert-for-token-classification"
n_samples = 500

subprocess.run(
"git sparse-checkout set examples/pytorch/token-classification",
shell=True,
cwd=os.path.join(self.dir_path, "transformers"),
)

subprocess.run(
f"python examples/pytorch/token-classification/run_ner.py"
f" --model_name_or_path {model_name}"
f" --dataset_name conll2003"
f" --do_eval"
f" --output_dir {os.path.join(self.dir_path, 'tokenclassification_conll2003_transformers')}"
f" --max_eval_samples {n_samples}",
shell=True,
cwd=os.path.join(self.dir_path, "transformers"),
)

> with open(
os.path.join(self.dir_path, "tokenclassification_conll2003_transformers", "eval_results.json"), "r"
) as f:
E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpg8va97zxevaluator_trainer_parity_test/tokenclassification_conll2003_transformers/eval_results.json'

tests/test_trainer_evaluator_parity.py:293: FileNotFoundError
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.