autogluon / autogluon/fev

Add "agentic" model type to submission metadata & leaderboard (similar to GIFT-Eval)

Open
#153 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
173
Forks
29
Avg merge
3h 19m
Merged PRs (30d)
10

Description

Add an `agentic` value to submission-level model-type metadata so agentic forecasting systems can self-declare and be filtered on the leaderboard. Metadata only. No change to tasks, metrics, EvaluationWindow, or aggregation.

**Motivation**
Agentic systems (tool-using, multi-step pipelines that produce forecasts) are now a distinct submission class, not plain pretrained or fine-tuned models. GIFT-Eval added an `Agentic` model type for this reason, and it gets used: the Moirai 2.0 paper filters out agentic solutions to get a clean foundation-model comparison. Without a tag, agentic submissions get excluded informally or silently skew a board readers assume is model v model.

**Approach**
- Extend the submission-level `model_type` field with an `agentic` value, surfaced as a filterable leaderboard column.
- Submission-level, not task-level, so it describes how the forecast was produced, orthogonal to the task taxonomy.
- Same scoring and statistical treatment as everything else. The tag only enables include or exclude, not a separate ranking.

Happy to send a PR for this.

Contributor guide

Open the contributing guide

Research direction

Locate the submission-level model_type definition and the leaderboard metadata and filtering entry points. Add support for the agentic value and verify that it appears as a filterable leaderboard category while task handling, metrics, EvaluationWindow, aggregation, scoring, and statistical treatment remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.