Feature Request: Enhanced Submission Analytics & Insights Dashboard
- Dominant language
- Python
- Stars
- 2k
- Forks
- 984
- Avg merge
- 2h 54m
- Merged PRs (30d)
- 14
Description
## Summary
The current analytics module in EvalAI provides only basic submission counts and participant metrics. This feature request proposes expanding the analytics capabilities to provide challenge hosts with deeper insights into submission patterns, evaluation performance, and participant engagement.
## Problem Statement
Currently, the analytics module (`apps/analytics/views.py`) offers limited functionality:
- Basic participant and team counts
- Simple submission counts (daily, weekly, monthly)
- Last submission timestamps
Challenge hosts lack visibility into:
- Submission success/failure trends
- Evaluation performance bottlenecks
- Participant engagement patterns over time
- Leaderboard dynamics and progression
## Proposed Features
### 1. Submission Success/Failure Rates per Phase
**Description:** Track and visualize the ratio of successful vs. failed submissions for each challenge phase.
**Metrics to include:**
- Success rate percentage per phase
- Failure breakdown by error type (timeout, runtime error, invalid format, etc.)
- Trend analysis over time (daily/weekly)
**API Endpoint Example:**
```
GET /api/analytics/challenges/{challenge_pk}/phases/{phase_pk}/submission-rates/
```
**Response Example:**
```json
{
"phase_id": 1,
"phase_name": "Test Phase",
"total_submissions": 1500,
"successful": 1200,
"failed": 300,
"success_rate": 80.0,
"failure_breakdown": {
"timeout": 120,
"runtime_error": 100,
"invalid_format": 50,
"other": 30
}
}
```
---
### 2. Average Evaluation Time Tracking
**Description:** Monitor and display average evaluation times to help hosts identify performance bottlenecks.
**Metrics to include:**
- Average evaluation time per phase
- 50th, 90th, 99th percentile evaluation times
- Evaluation time trends over the past 7/30 days
- Comparison across different phases
**API Endpoint Example:**
```
GET /api/analytics/challenges/{challenge_pk}/evaluation-times/
```
**Response Example:**
```json
{
"challenge_id": 1,
"phases": [
{
"phase_id": 1,
"phase_name": "Dev Phase",
"avg_evaluation_time_seconds": 45.2,
"p50": 30.0,
"p90": 120.0,
"p99": 300.0
}
],
"trend": [
{"date": "2026-01-10", "avg_time": 42.5},
{"date": "2026-01-11", "avg_time": 48.3}
]
}
```
---
### 3. Participant Activity Heatmaps
**Description:** Visualize when participants are most active to help hosts optimize challenge scheduling and support availability.
**Metrics to include:**
- Submissions by hour of day (UTC)
- Submissions by day of week
- Geographic distribution of activity (if available)
- Peak activity periods
**API Endpoint Example:**
```
GET /api/analytics/challenges/{challenge_pk}/activity-heatmap/
```
**Response Example:**
```json
{
"challenge_id": 1,
"heatmap": {
"Monday": {"00": 5, "01": 3, "02": 2, ..., "23": 12},
"Tuesday": {"00": 4, "01": 2, ..., "23": 15}
},
"peak_hours": ["14:00-16:00 UTC", "20:00-22:00 UTC"],
"most_active_day": "Saturday"
}
```
---
### 4. Leaderboard Progression Charts
**Description:** Track how the leaderboard evolves over time, showing score improvements and ranking changes.
**Metrics to include:**
- Top-N teams' score progression over time
- Rank changes history for specific teams
- Best score improvements per time period
- New entries to top-N tracking
**API Endpoint Example:**
```
GET /api/analytics/challenges/{challenge_pk}/phases/{phase_pk}/leaderboard-progression/
```
**Response Example:**
```json
{
"phase_id": 1,
"top_teams_progression": [
{
"team_name": "Team Alpha",
"progression": [
{"date": "2026-01-01", "score": 0.85, "rank": 3},
{"date": "2026-01-05", "score": 0.89, "rank": 1}
]
}
],
"score_improvements": {
"biggest_jump": {"team": "Team Beta", "improvement": 0.15},
"avg_improvement": 0.03
}
}
```
---
## Technical Implementation
### Backend Changes
1. **New Models** (optional):
- `SubmissionAnalytics` - Store aggregated analytics data
- `EvaluationMetrics` - Track evaluation performance
2. **New Serializers** in `apps/analytics/serializers.py`:
- `SubmissionRateSerializer`
- `EvaluationTimeSerializer`
- `ActivityHeatmapSerializer`
- `LeaderboardProgressionSerializer`
3. **New Views** in `apps/analytics/views.py`:
- `get_submission_success_rates()`
- `get_evaluation_time_metrics()`
- `get_participant_activity_heatmap()`
- `get_leaderboard_progression()`
4. **Database Queries:**
- Leverage Django ORM aggregation functions
- Consider caching for frequently accessed analytics
- Use background tasks (Celery) for heavy computations
### Frontend Changes
1. **New Components** in `frontend_v2/src/app/components/`:
- `analytics-dashboard/`
- `submission-rate-chart/`
- `evaluation-time-chart/`
- `activity-heatmap/`
- `leaderboard-progression/`
2. **Visualization Libraries:**
- Use Chart.js or D3.js for interactive charts
- Heatmap.js for activity heatmaps
---
## Mockups
### Submission Success/Failure Rates
```
┌─────────────────────────────────────────┐
│ Dev Phase - Submission Results │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │
│ ████████████████████░░░░░ 80% Success │
│ │
│ Failed: 300 (20%) │
│ ├── Timeout: 120 │
│ ├── Runtime Error: 100 │
│ └── Invalid Format: 80 │
└─────────────────────────────────────────┘
```
### Activity Heatmap
```
Mon Tue Wed Thu Fri Sat Sun
00:00 ░ ░ ░ ░ ░ █ █
06:00 ░ ░ ░ ░ ░ ▓ ▓
12:00 ▓ ▓ ▓ ▓ ▓ █ █
18:00 █ █ █ █ █ ▓ ▓
░ Low ▓ Medium █ High
```
---
## Benefits
- **For Challenge Hosts:**
- Better understanding of participant behavior
- Identify evaluation bottlenecks
- Optimize challenge phase timings
- Data-driven decision making
- **For Participants:**
- Understand their progress relative to the leaderboard
- Know optimal times to submit
- **For Platform:**
- Differentiate from competitors
- Increase host engagement and retention
Contributor guide
Research direction
Start with the existing analytics module in apps/analytics/views.py and apps/analytics/serializers.py, then review the current endpoints and data available for submissions, evaluations, and leaderboards. Inspect frontend_v2/src/app/components/ alongside the proposed chart components and compare the available Chart.js or D3.js integration. Done means the requested analytics metrics are available through documented endpoints and represented in the dashboard.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- chart.js, django, python
- Domain
- analytics, backend, frontend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100