dotnet-ai evaluation OOM-killed on standard runners
- Dominant language
- C#
- Stars
- 5.4k
- Forks
- 415
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 81
Description
## `dotnet-ai` evaluation OOM-killed on standard runners
### Problem
The `dotnet-ai` plugin evaluation is consistently being killed by the OOM killer (exit code 137) on `ubuntu-latest` runners. This was first observed in scheduled run [23070747448](https://github.com/dotnet/skills/actions/runs/23070747448/job/67020509745).
The single skill in this plugin — `technology-selection` — has scenarios that build and run ML.NET projects (`TicketClassifier`, `ChurnPredictor`). With the scheduled evaluation parallelism settings (`--parallel-scenarios 3 --parallel-runs 3`), up to 9 concurrent agent runs can each `dotnet build` + `dotnet run` these ML.NET projects simultaneously, each loading training data and building models in memory. This exceeds the runner's ~7 GB RAM limit.
At the time of the OOM kill, the orphan process cleanup shows:
- 5x `TicketClassifier` processes
- 3x `ChurnPredictor` processes
- ~20 bash shells (agent sandboxes)
The scenarios responsible are:
- **"ML.NET classification on tabular data"** — builds and runs `TicketClassifier`
- **"Reject LLM for tabular classification"** — builds and runs `ChurnPredictor`
### Impact
- `dotnet-ai` evaluation never completes successfully
- Prior to #365, this also blocked `publish-benchmark` for **all** plugins, preventing any dashboard updates since March 11
- No evaluation data for `dotnet-ai` appears on the dashboard
### Suggested fix
The simplest fix would be to reduce the parallelism for these ML.NET scenarios. Some options (in order of increasing effort):
1. **Reduce scenario count in eval.yaml**: If some of the 6 scenarios are lower priority, reducing to 3-4 would cut peak memory.
2. **Make scenarios lighter**: Use smaller training datasets (`tickets.csv`, `customers.csv`) so the ML.NET models consume less memory during training/inference.
3. **Avoid `dotnet run` in scenario setup**: If the scenarios only need the agent to write the code (not execute it), removing the runtime step would eliminate the ML.NET memory overhead entirely. The agent can still `dotnet build` to verify compilation.
4. **Request lower parallelism for this plugin**: The workflow could be extended to support per-plugin parallelism overrides. For example, `dotnet-ai` could declare `"maxParallelScenarios": 1` in its `plugin.json`.
cc @jeffschwMSFT @luisquintanilla — you authored the `technology-selection` skill in #177. Would any of the above approaches work for you?
### Reproduction
The failure occurs reliably on every scheduled evaluation run since `dotnet-ai` was added to the repo. See:
- Run [23070747448](https://github.com/dotnet/skills/actions/runs/23070747448) — `evaluate (dotnet-ai)` killed with exit 137
### Environment
- Runner: `ubuntu-latest` (standard GitHub-hosted, ~7 GB RAM)
- Job timeout: 90 minutes
- Parallelism: `--parallel-skills 2 --parallel-scenarios 3 --parallel-runs 3`
- Exit code: 137 (SIGKILL / OOM)
Contributor guide
Research direction
Start with eval.yaml, the technology-selection scenarios, and the TicketClassifier and ChurnPredictor projects named in the issue. Reproduce the dotnet-ai evaluation with the documented parallelism on ubuntu-latest, then choose and validate an approach that prevents the runner from exceeding its memory limit. Done means the evaluation completes without exit code 137 and produces dashboard data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, github-actions
- Domain
- ci-cd, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100