dice-group / dice-group/gerbil

Datasets with more than 1 question don't work

Open
#399 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
231
Forks
55
PR merge metrics
No merged PRs in 30d

Description

I try to use my own dataset while evaluating a QA system, however, after I configure and start the experiment nothing happens -- "The experiment is still running.", but the system doesn't receive the questions.

Occasionally, I tried to upload the dataset with only one question in it -- it worked, but I still want to evaluate on all the data I have.

I use this endpoint: http://gerbil-qa.aksw.org/gerbil/config

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the problem at the reported configuration endpoint with datasets containing one and multiple questions, and compare the experiment status and system behavior. Trace the dataset upload and experiment execution path to find where multi-question input stops progressing; done means all questions are delivered and the experiment completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.