Multiple choice datasets encounter a tensor error
- Dominant language
- Python
- Stars
- 225
- Forks
- 30
- PR merge metrics
- No merged PRs in 30d
Description
Multiple choice datasets such as "glue:mnli" returns a tensor error.
The command
`elk elicit gpt2 "glue:mnli" --num_gpus 2`
returns this output
> Using 2 of 8 GPUs: [0, 4]
> glue:mnli using 'train' for training and 'validation_matched' for validation
> Found cached dataset generator (/home/ethan/.cache/huggingface/datasets/generator/default-a729374a01df289c/0.0.0)
> Found cached dataset generator (/home/ethan/.cache/huggingface/datasets/generator/default-5453548fe42aba36/0.0.0)
> Output directory at /home/ethan/elk-reporters/gpt2/glue:mnli/kind-golick
> 0%| | 0/13 [00:08 multiprocessing.pool.RemoteTraceback:
> """
> Traceback (most recent call last):
> File "/mnt/ssd-1/spar/ethan/miniconda3/lib/python3.10/multiprocessing/pool.py", line 125, in worker
> result = (True, func(*args, **kwds))
> File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/training/train.py", line 133, in apply_to_layer
> **evaluate_preds(val_gt, val_credences, mode).to_dict(),
> File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/metrics/eval.py", line 75, in evaluate_preds
> auroc = roc_auc_ci(to_one_hot(y_true, c).long(), y_logits)
> File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/metrics/roc_auc.py", line 103, in roc_auc_ci
> raise ValueError("y_true and y_pred should be 1D or 2D tensors")
> ValueError: y_true and y_pred should be 1D or 2D tensors
> """
>
> The above exception was the direct cause of the following exception:
>
> Traceback (most recent call last):
> File "/mnt/ssd-1/spar/ethan/miniconda3/bin/elk", line 8, in
> sys.exit(run())
> File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/__main__.py", line 27, in run
> run.execute()
> File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/__main__.py", line 19, in execute
> return self.command.execute()
> File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/run.py", line 98, in execute
> self.apply_to_layers(func=func, num_devices=num_devices)
> File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/run.py", line 175, in apply_to_layers
> for df_dict in tqdm(mapper(func, layers), total=len(layers)):
> File "/mnt/ssd-1/spar/ethan/miniconda3/lib/python3.10/site-packages/tqdm/std.py", line 1195, in __iter__
> for obj in iterable:
> File "/mnt/ssd-1/spar/ethan/miniconda3/lib/python3.10/multiprocessing/pool.py", line 873, in next
> raise value
> ValueError: y_true and y_pred should be 1D or 2D tensors
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.