Add "agentic" model type to submission metadata & leaderboard (similar to GIFT-Eval)
- Dominant language
- Python
- Stars
- 173
- Forks
- 29
- Avg merge
- 3h 19m
- Merged PRs (30d)
- 10
Description
Add an `agentic` value to submission-level model-type metadata so agentic forecasting systems can self-declare and be filtered on the leaderboard. Metadata only. No change to tasks, metrics, EvaluationWindow, or aggregation.
**Motivation**
Agentic systems (tool-using, multi-step pipelines that produce forecasts) are now a distinct submission class, not plain pretrained or fine-tuned models. GIFT-Eval added an `Agentic` model type for this reason, and it gets used: the Moirai 2.0 paper filters out agentic solutions to get a clean foundation-model comparison. Without a tag, agentic submissions get excluded informally or silently skew a board readers assume is model v model.
**Approach**
- Extend the submission-level `model_type` field with an `agentic` value, surfaced as a filterable leaderboard column.
- Submission-level, not task-level, so it describes how the forecast was produced, orthogonal to the task taxonomy.
- Same scoring and statistical treatment as everything else. The tag only enables include or exclude, not a separate ranking.
Happy to send a PR for this.
Contributor guide
Research direction
Locate the submission-level model_type definition and the leaderboard metadata and filtering entry points. Add support for the agentic value and verify that it appears as a filterable leaderboard category while task handling, metrics, EvaluationWindow, aggregation, scoring, and statistical treatment remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100