dice-group / dice-group/gerbil
[QA] Leaderboard for challenges is too slow
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 231
- Forks
- 55
- PR merge metrics
- No merged PRs in 30d
Description
Problem Description
The overview page doesn't seem to work anymore. If data is requested, the requests seems to be running forever.
Analysis
The problem is caused by the leaderboard, which has been introduced for the QALD challenge. The way the leaderboard is generated (for each language and dataset, search for the best performance for each system) becomes too expensive for the data that we currently have in our database (22 languages, 35 datasets and 33797 systems).
Hotfix
- Disable the leaderboard
Solution
List ExperimentTask IDs that are relevant for the leaderboard of a challenge in an additional table
- When GERBIL is started with a challenge configured, it should run a
CREATE TABLE IF NOT EXISTSquery to create leaderboard table - When an experiment finishes, a simple check whether the result is better than the best result of this system in the current leaderboard can be used to update the leaderboard if necessary
- When the leaderboard is requested, only the experiment tasks listed in the leaderboard table have to be retrieved.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by tracing GERBIL's challenge startup, experiment completion, and leaderboard request paths, then identify where the current per-language and per-dataset search runs. Done means the leaderboard uses a maintained table of relevant ExperimentTask IDs and the overview request no longer performs the expensive full search.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- database, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100