dice-group / dice-group/gerbil
Datasets with more than 1 question don't work
- Dominant language
- Java
- Stars
- 231
- Forks
- 55
- PR merge metrics
- No merged PRs in 30d
Description
I try to use my own dataset while evaluating a QA system, however, after I configure and start the experiment nothing happens -- "The experiment is still running.", but the system doesn't receive the questions.
Occasionally, I tried to upload the dataset with only one question in it -- it worked, but I still want to evaluate on all the data I have.
I use this endpoint: http://gerbil-qa.aksw.org/gerbil/config
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the problem at the reported configuration endpoint with datasets containing one and multiple questions, and compare the experiment status and system behavior. Trace the dataset upload and experiment execution path to find where multi-question input stops progressing; done means all questions are delivered and the experiment completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100