huggingface / huggingface/evaluate
Concurrent Programs Sharing Evaluate Lib Would raise "ValueError: Error in finalize: another evaluation module instance is already using the local cache file."
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
I have a similar problem to #481. I am running 10 independent experiments together on different GPU but same file system, which is implemented by LSF (similar to SLURM). And then `metric.compute` have exceptions because multi experiment processes sharing the same code but different parameters, conflicts.
### The code that I use evaluate is like:
```python
exp_id = os.environ['exp_id'] or 0 # I can have this, but I don't know how to pass this to metric
metric = evaluate.load("accuracy")
def compute_metrics(eval_pred):
"""Computes accuracy on a batch of predictions"""
predictions = np.argmax(eval_pred.predictions, axis=1)
return metric.compute(predictions=predictions, references=eval_pred.label_ids)
```
### The error is as follows:
```bash
0%| | 0/1 [00:00
train(
File "/home/researches/hugging_face/hugging_face/src/commons.py", line 137, in train
train_results = trainer.train()
File "/home/miniconda3/envs/hf_ai/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
return inner_training_loop(
File "/home/miniconda3/envs/hf_ai/lib/python3.10/site-packages/accelerate/utils/memory.py", line 136, in decorator
return function(batch_size, *args, **kwargs)
File "/home/miniconda3/envs/hf_ai/lib/python3.10/site-packages/transformers/trainer.py", line 1916, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
File "/home/miniconda3/envs/hf_ai/lib/python3.10/site-packages/transformers/trainer.py", line 2226, in _maybe_log_save_evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "/home/miniconda3/envs/hf_ai/lib/python3.10/site-packages/transformers/trainer.py", line 2934, in evaluate
output = eval_loop(
File "/home/miniconda3/envs/hf_ai/lib/python3.10/site-packages/transformers/trainer.py", line 3222, in evaluation_loop
metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
File "/home/researches/hugging_face/hugging_face/src/commons.py", line 109, in compute_metrics
return metric.compute(predictions=predictions, references=eval_pred.label_ids)
File "/home/miniconda3/envs/hf_ai/lib/python3.10/site-packages/evaluate/module.py", line 433, in compute
self._finalize()
File "/home/miniconda3/envs/hf_ai/lib/python3.10/site-packages/evaluate/module.py", line 392, in _finalize
raise ValueError(
ValueError: Error in finalize: another evaluation module instance is already using the local cache file. Please specify an experiment_id to avoid collision between distributed evaluation module instances.
28%|██▊ | 843/3000 [1:05:13<2:46:54, 4.64s/it]
```
The problem is on this code `evaluate/module.py` line 392
```python
elif self.process_id == 0:
# Let's acquire a lock on each node files to be sure they are finished writing
file_paths, filelocks = self._get_all_cache_files()
# Read the predictions and references
try:
reader = ArrowReader(path="", info=DatasetInfo(features=self.selected_feature_format))
self.data = Dataset(**reader.read_files([{"filename": f} for f in file_paths]))
except FileNotFoundError:
raise ValueError(
"Error in finalize: another evaluation module instance is already using the local cache file. "
"Please specify an experiment_id to avoid collision between distributed evaluation module instances."
) from None
# Store file paths and locks and we will release/delete them after the computation.
self.file_paths = file_paths
self.filelocks = filelocks
```
Contributor guide
Assessment
This issue has not been assessed yet.