huggingface / huggingface/transfer-learning-conv-ai

Error when setting num_candidates=1 and in dataset there's only one candidate during validation

Open
#40 7 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.8k
Forks
430
PR merge metrics
No merged PRs in 30d

Description

When setting args.num_candidates to 1, and the actual length of the candidates list of each entry is 1, I get this error during validation:
```
ERROR:ignite.engine.engine.Engine:Current run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's..
ERROR:ignite.engine.engine.Engine:Engine run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's..
ERROR:ignite.engine.engine.Engine:Engine run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's..
Traceback (most recent call last):
File "./train-regular.py", line 277, in
train()
File "./train-regular.py", line 269, in train
trainer.run(train_loader, max_epochs=args.n_epochs)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 446, in run
self._handle_exception(e)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 410, in _handle_exception
raise e
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 437, in run
self._fire_event(Events.EPOCH_COMPLETED)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 345, in _fire_event
func(self, *(event_args + args), **kwargs)
File "./train-regular.py", line 223, in
trainer.add_event_handler(Events.EPOCH_COMPLETED, lambda _: evaluator.run(val_loader))
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 446, in run
self._handle_exception(e)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 410, in _handle_exception
raise e
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 433, in run
hours, mins, secs = self._run_once_on_dataset()
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 399, in _run_once_on_dataset
self._handle_exception(e)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 410, in _handle_exception
raise e
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 392, in _run_once_on_dataset
self._fire_event(Events.ITERATION_COMPLETED)
File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 345, in _fire_event
func(self, *(event_args + args), **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
return func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/metric.py", line 65, in iteration_completed
self.update(output)
File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/accuracy.py", line 126, in update
self._check_type((y_pred, y))
File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/accuracy.py", line 57, in _check_type
self._check_binary_multilabel_cases((y_pred, y))
File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/accuracy.py", line 48, in _check_binary_multilabel_cases
raise ValueError("For binary cases, y_pred must be comprised of 0's and 1's.")
ValueError: For binary cases, y_pred must be comprised of 0's and 1's.
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure with args.num_candidates set to 1 and inspect train-regular.py, especially the evaluator setup around line 223 and the training entry point around line 269. Trace the validation output passed to Ignite's Accuracy metric and verify that validation completes successfully for entries with one candidate, with the metric receiving valid binary predictions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.