microsoft / microsoft/ebpf-for-windows
Add Azure Function to Analyze Logs with LLM for First-Pass Workflow Failure Triage
- Dominant language
- C
- Stars
- 3.6k
- Forks
- 311
- Avg merge
- 6d 10h
- Merged PRs (30d)
- 21
Description
**Feature request summary**
Today, when workflows fail in eBPF for Windows CI, the logging output is extremely shallow. The logs primarily contain basic task-level information, making it difficult to identify root causes or patterns in recurring failures. This slows down both triage and reliability efforts.
**Ask**
Add an **Azure Function–based log analysis pipeline** that automatically processes workflow logs through an **LLM** and produces a **first‑pass triage summary**. This summary should be attached to the workflow run (e.g., as an artifact, PR comment, or GitHub Check output).
**Motivation**
* CI failures are becoming more frequent and more complex.
* Many failures follow known patterns, but the raw logs provide little structured signal.
* Human triage time is high, and logs often need several minutes of manual digging.
* GitHub Copilot and LLMs are well‑suited to classify likely failure categories (infra vs environment vs build vs test vs flake) and extract contributing factors.
**Proposed design**
1. **Trigger:**
* Workflow completion event (success/failure) or only on failure.
2. **Data collection:**
* Retrieve workflow logs via GitHub Actions API.
* Compress and upload to an input queue or storage binding for the Azure Function.
3. **Azure Function:**
* Stateless function (Python or C#) that loads logs, performs chunking, and runs them through an LLM.
* Compatible with Azure OpenAI, OnnxRuntime-GenAI, or other compliant provider.
* Produces JSON output:
```json
{
"summary": "...",
"likely_cause": "...",
"confidence": 0.0 - 1.0,
"related_components": ["..."],
"suggested_next_steps": ["..."],
"possible_fix_locations": ["..."]
}
```
4. **Output surfacing:**
* Publish as a GitHub Check summary, artifact, or pull request comment.
* Optionally store structured triage data for metrics.
**Example benefits**
* Faster triage of intermittent or environment-related failures.
* Reduced developer time wasted on parsing logs manually.
* Consistent classification of recurring failures (build timeout, NuGet restore issues, test flaky behavior, environment instability, etc.).
* Could enable later automated grouping of failures across runs.
**Additional considerations**
* Add privacy/filtering layer (strip secrets/tokens).
* Include a fallback check if the LLM is unavailable.
* Ensure logs stay within token limits via chunking/summary folding.
**Request for feedback**
Looking for agreement on:
* The idea itself
* Whether the function should run on all failures or only specific workflows
* Preferred placement (this repo vs infra repo)
* Whether to use Azure Functions, Container Apps, or GitHub Actions self‑hosted runner for inference
Contributor guide
Assessment
This issue has not been assessed yet.