tensorflow / tensorflow/datasets
QASC dataset: answerKey missing in test split
Open
Nobody has claimed this yet.
bug
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Short description
In QASC dataset, answerKey field is missing in the test samples.
Environment information
- Operating System: Ubuntu
- Python version: 3.7
tensorflow-datasets/tfds-nightlyversion: 3.2.1tensorflow/tf-nightlyversion: 3.2.1.dev202009090105
Reproduction instructions
import tensorflow_datasets as tfds
import tensorflow as tf
tf.enable_eager_execution()
ds = tfds.load('qasc', split="test", shuffle_files=True)
for datapoint in ds:
facts = [datapoint["fact1"].numpy().decode("utf-8"), datapoint["fact2"].numpy().decode("utf-8")]
question = datapoint["question"].numpy().decode("utf-8")
choices = {label.decode("utf-8"): text.decode("utf-8")
for label, text in zip(datapoint["choices"]["label"].numpy(),
datapoint["choices"]["text"].numpy())}
answer = datapoint["answerKey"].numpy().decode("utf-8")
print(answer)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue by loading the QASC test split with the provided Python snippet and inspecting the dataset definition for the test samples. Check whether answerKey is included in the exposed test features, then rerun the snippet to confirm that every test sample provides it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100