dice-group / dice-group/gerbil

[QA] Leaderboard for challenges is too slow

Open
#442 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

priority:medium type:bug type:enhancement
Dominant language
Java
Stars
231
Forks
55
PR merge metrics
No merged PRs in 30d

Description

Problem Description

The overview page doesn't seem to work anymore. If data is requested, the requests seems to be running forever.

Analysis

The problem is caused by the leaderboard, which has been introduced for the QALD challenge. The way the leaderboard is generated (for each language and dataset, search for the best performance for each system) becomes too expensive for the data that we currently have in our database (22 languages, 35 datasets and 33797 systems).

Hotfix

  • Disable the leaderboard

Solution

List ExperimentTask IDs that are relevant for the leaderboard of a challenge in an additional table

  • When GERBIL is started with a challenge configured, it should run a CREATE TABLE IF NOT EXISTS query to create leaderboard table
  • When an experiment finishes, a simple check whether the result is better than the best result of this system in the current leaderboard can be used to update the leaderboard if necessary
  • When the leaderboard is requested, only the experiment tasks listed in the leaderboard table have to be retrieved.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by tracing GERBIL's challenge startup, experiment completion, and leaderboard request paths, then identify where the current per-language and per-dataset search runs. Done means the leaderboard uses a maintained table of relevant ExperimentTask IDs and the overview request no longer performs the expensive full search.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.