microsoft / microsoft/ebpf-for-windows

Add Azure Function to Analyze Logs with LLM for First-Pass Workflow Failure Triage

Open
#4,990 1 comment 0 reactions 1 assignee Claimed by @nmlud21 View on GitHub
ci/cd triaged
Dominant language
C
Stars
3.6k
Forks
311
Avg merge
6d 10h
Merged PRs (30d)
21

Description

**Feature request summary**
Today, when workflows fail in eBPF for Windows CI, the logging output is extremely shallow. The logs primarily contain basic task-level information, making it difficult to identify root causes or patterns in recurring failures. This slows down both triage and reliability efforts.

**Ask**
Add an **Azure Function–based log analysis pipeline** that automatically processes workflow logs through an **LLM** and produces a **first‑pass triage summary**. This summary should be attached to the workflow run (e.g., as an artifact, PR comment, or GitHub Check output).

**Motivation**

* CI failures are becoming more frequent and more complex.
* Many failures follow known patterns, but the raw logs provide little structured signal.
* Human triage time is high, and logs often need several minutes of manual digging.
* GitHub Copilot and LLMs are well‑suited to classify likely failure categories (infra vs environment vs build vs test vs flake) and extract contributing factors.

**Proposed design**

1. **Trigger:**
* Workflow completion event (success/failure) or only on failure.

2. **Data collection:**
* Retrieve workflow logs via GitHub Actions API.
* Compress and upload to an input queue or storage binding for the Azure Function.

3. **Azure Function:**
* Stateless function (Python or C#) that loads logs, performs chunking, and runs them through an LLM.
* Compatible with Azure OpenAI, OnnxRuntime-GenAI, or other compliant provider.
* Produces JSON output:
```json
{
"summary": "...",
"likely_cause": "...",
"confidence": 0.0 - 1.0,
"related_components": ["..."],
"suggested_next_steps": ["..."],
"possible_fix_locations": ["..."]
}
```

4. **Output surfacing:**
* Publish as a GitHub Check summary, artifact, or pull request comment.
* Optionally store structured triage data for metrics.

**Example benefits**

* Faster triage of intermittent or environment-related failures.
* Reduced developer time wasted on parsing logs manually.
* Consistent classification of recurring failures (build timeout, NuGet restore issues, test flaky behavior, environment instability, etc.).
* Could enable later automated grouping of failures across runs.

**Additional considerations**

* Add privacy/filtering layer (strip secrets/tokens).
* Include a fallback check if the LLM is unavailable.
* Ensure logs stay within token limits via chunking/summary folding.

**Request for feedback**
Looking for agreement on:

* The idea itself
* Whether the function should run on all failures or only specific workflows
* Preferred placement (this repo vs infra repo)
* Whether to use Azure Functions, Container Apps, or GitHub Actions self‑hosted runner for inference

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.