[Feature]: Implement scheduled report execution and proactive anomaly monitoring

### What problem are you trying to solve?

The Azure FinOps agent is currently entirely reactive — it only analyzes data when a user types a question in the chat. Users in large production tenants have asked for proactive capabilities: the agent should discover problems, detect anomalies, and generate reports on a schedule without requiring the user to be present. Today, when a user wants a daily cost anomaly check or a weekly waste scan, they must manually open the browser, type the question, wait for the query to complete, and review the results. This doesn't scale.

**Root cause analysis based on the current codebase:**

1. **`ScheduleTools.cs` stores schedule metadata but nothing executes them** — the existing `SaveReportSchedule`, `ListReportSchedules`, and `DeleteReportSchedule` tools persist schedule definitions (prompt, frequency, scope) to `~/finops-agent-schedules/report-schedules.json`. The class exposes a `GetAll()` method explicitly designed for a background service consumer. But no background service exists to read these schedules and execute them. The code comment says _"A background service could pick these up"_ — this issue implements that.

2. **All tool execution requires an active HTTP request** — `ChatEndpoints.cs` invokes tools through the Copilot SDK `session.SendAsync()` call, which is always triggered by a user's `POST /api/chat`. There is no pathway to invoke tools from a background service or timer. The `CopilotSessionFactory` creates sessions per-user with user-scoped tokens, and cannot currently create a "system" session for background work.

3. **Anomaly detection (`AnomalyTools.cs`) is on-demand only** — the tool computes z-score anomalies against a rolling baseline and returns structured JSON. It is fully functional for background use (takes a subscription ID, returns a self-contained result), but is only invoked when a user asks. Cost spikes that happen overnight or on weekends go undetected until someone manually checks.

4. **Idle resource scanning (`IdleResourceTools.cs`) runs only on user request** — the waste scan queries 8 Resource Graph patterns (unattached disks, orphan IPs, stopped VMs, etc.) and returns a consolidated report. This is an ideal candidate for scheduled background execution, but has no scheduling infrastructure.

5. **FinOps maturity scoring (`ScoreTools.cs`) has history tracking but no automated re-assessment** — scores are persisted to `~/finops-agent-scores/score-history.json` with timestamps for trend analysis. But re-scoring only happens when the user explicitly asks. Monthly automated re-scoring would enable regression detection.

6. **No notification delivery mechanism** — even if background tasks ran, there is no way to surface results to users. No notification inbox, no badge/indicator, no email integration, and no webhook support.

**Dependency:** This issue builds on the persistent user sessions issue. Deterministic Entra ID-based user identity (Layer 1) is required so that scheduled tasks can be associated with a specific user and their results can be delivered to the correct inbox. Redis-backed sessions (Layer 3) ensure that the scheduler service survives container restarts.

### Proposed solution

A three-layer approach: a background scheduler service, a notification inbox for results delivery, and direct Azure API tool execution without the LLM for predictable scheduled tasks.

### Layer 1: Background scheduler service (`ScheduleRunnerService`)

**New file:** `src/Dashboard/Services/ScheduleRunnerService.cs`

Create a new `BackgroundService` following the existing `UserStateJanitor` pattern. This service runs on a timer, reads saved schedules from `ScheduleTools.GetAll()`, determines which are due, and executes them.

Implementation:

- Register in `Program.cs` via `builder.Services.AddHostedService()`.
- Run the evaluation loop every **5 minutes** (configurable). On each tick:
1. Call `ScheduleTools.GetAll()` to get all enabled schedules.
2. For each schedule, check if it is due based on `Frequency` and `LastRunUtc`:
- `daily`: due if `LastRunUtc` is null or >24 hours ago
- `weekly`: due if `LastRunUtc` is null or >7 days ago
- `monthly`: due if `LastRunUtc` is null or the calendar month has changed
3. Execute due schedules sequentially (not in parallel — avoids thundering herd on Azure APIs).
4. After execution, update `LastRunUtc` on the schedule via a new `ScheduleTools.UpdateLastRun(id)` method.
- Wrap each execution in a try/catch — a failing schedule must not block other schedules.
- Log execution start/completion/failure via OpenTelemetry (`AiTelemetry.ActivitySource`).
- Add a new counter metric: `finops.schedule.runs` with tags `schedule_id`, `frequency`, `status` (success/failure).

**Execution strategy — direct tool invocation, not LLM:**

For scheduled tasks, bypass the Copilot SDK / LLM entirely. The LLM adds latency, cost ($0.01–0.10 per invocation), and unpredictability. Scheduled tasks have fixed, known prompts that map to specific tools. Execute the tools directly:

```csharp
private async Task ExecuteSchedule(ScheduleTools.ReportSchedule schedule, CancellationToken ct)
{
// Get the system service token (managed identity) for Azure ARM
var credential = new DefaultAzureCredential();
var tokenResult = await credential.GetTokenAsync(
new TokenRequestContext(new[] { "https://management.azure.com/.default" }), ct);

var systemTokens = new UserTokens { AzureToken = tokenResult.Token };

// Route by schedule type — no LLM needed for predictable queries
string result;
if (schedule.Prompt.Contains("anomal", StringComparison.OrdinalIgnoreCase)
|| schedule.Prompt.Contains("spike", StringComparison.OrdinalIgnoreCase))
{
var anomalyTools = new AnomalyTools(systemTokens);
result = await InvokeToolByName(anomalyTools, "DetectCostAnomalies", schedule);
}
else if (schedule.Prompt.Contains("idle", StringComparison.OrdinalIgnoreCase)
|| schedule.Prompt.Contains("waste", StringComparison.OrdinalIgnoreCase)
|| schedule.Prompt.Contains("orphan", StringComparison.OrdinalIgnoreCase))
{
var idleTools = new IdleResourceTools(systemTokens);
result = await InvokeToolByName(idleTools, "FindIdleResources", schedule);
}
else
{
// Generic schedule — store the prompt for next user session
result = $"Schedule '{schedule.Name}' is due. Prompt: {schedule.Prompt}";
}

// Save result to notification store
NotificationStore.Add(schedule.UserId, new Notification
{
Id = Guid.NewGuid().ToString("N")[..8],
ScheduleId = schedule.Id,
ScheduleName = schedule.Name,
CreatedUtc = DateTime.UtcNow,
Summary = ExtractSummary(result),
FullResult = result,
Read = false
});
}
```

Authentication for background execution:
- Use **Managed Identity** (`DefaultAzureCredential`) for system-level ARM queries. This requires granting the App Service / Container App's managed identity `Reader` role on the target subscriptions.
- This is separate from the user's delegated OAuth tokens — background tasks run with application-level permissions.
- Document in README that managed identity must be configured and granted access for scheduled features to work.

### Layer 2: Notification store and API

**New file:** `src/Dashboard/Services/NotificationStore.cs`

A lightweight in-memory + file-persisted store for schedule results, scoped per user.

Implementation:

- Storage: `~/finops-agent-notifications/{userId}/` directory with one JSON file per notification.
- Data model:
```csharp
public class Notification
{
public string Id { get; set; }
public string ScheduleId { get; set; }
public string ScheduleName { get; set; }
public DateTime CreatedUtc { get; set; }
public string Summary { get; set; } // 1-2 sentence headline
public string FullResult { get; set; } // Complete tool output JSON
public bool Read { get; set; }
public string? Severity { get; set; } // "info", "warning", "critical"
}
```
- Retention: auto-delete notifications older than **30 days**. Run cleanup in the same `ScheduleRunnerService` timer loop.
- Cap: maximum **100 notifications per user**. Drop oldest when exceeded.

**New file:** `src/Dashboard/Endpoints/NotificationEndpoints.cs`

API surface for the frontend:

```
GET /api/notifications — list notifications for current user (newest first, max 50)
GET /api/notifications/unread — count of unread notifications (for badge)
POST /api/notifications/{id}/read — mark a notification as read
POST /api/notifications/{id}/dismiss — delete a notification
POST /api/notifications/read-all — mark all as read
```

All endpoints require an authenticated session (check `ctx.Session.GetString("user")`). Notifications are scoped by the deterministic Entra OID-derived userId from the persistent sessions issue.

### Layer 3: Frontend notification indicator and inbox

**File:** `src/Dashboard/frontend/src/components/ChatView.vue`

Add a notification bell icon to the header/sidebar that shows unread count and allows viewing results.

Implementation:

- On component mount and every **60 seconds**, poll `GET /api/notifications/unread` to get the unread count. Display as a badge on a bell icon in the sidebar header.
- Clicking the bell opens a notification dropdown/panel listing recent notifications with: schedule name, timestamp, severity icon, and 1-line summary.
- Clicking a notification marks it as read (`POST /api/notifications/{id}/read`) and injects the full result into the chat as a system message — the same rendering pipeline used for regular assistant responses (markdown, charts, tables).
- Add a "Mark all read" action in the dropdown header.
- No WebSocket/SSE push required — 60-second polling is sufficient for background reports that run hourly/daily/weekly. This avoids the complexity of persistent connections.

### Layer 4: Extend `ScheduleTools` to support user binding and schedule types

**File:** `src/Dashboard/AI/Tools/ScheduleTools.cs`

Extend the existing `ReportSchedule` model and tools to support the new execution infrastructure.

Implementation:

- Add fields to `ReportSchedule`:
```csharp
public long UserId { get; set; } // Entra OID-derived deterministic ID (from sessions issue)
public string ScheduleType { get; set; } // "anomaly_check", "waste_scan", "maturity_score", "custom"
public string? SubscriptionIds { get; set; } // Comma-separated target subscriptions
public string? NotifySeverity { get; set; } // "all", "warning_and_critical", "critical_only"
```
- Update `SaveReportSchedule` to accept `userId` from the session context (pass through from `ChatEndpoints`). The tool currently has no access to the user identity — it needs to be provided via constructor injection or a parameter.
- Add a new `UpdateLastRun(string id, DateTime lastRunUtc)` method for the background service to update after execution.
- Add a `GetDueSchedules(DateTime now)` method that encapsulates the due-check logic (avoids duplicating it in the background service).
- Migrate storage from single `report-schedules.json` to per-user files: `~/finops-agent-schedules/{userId}/schedules.json`. This ensures users can only see and manage their own schedules.

### Built-in schedule templates

Add pre-defined schedule types that map to specific tool invocations, avoiding the need for LLM interpretation:

| Schedule Type | Tool | Default Frequency | Default Severity |
|--------------|------|-------------------|-----------------|
| `anomaly_check` | `AnomalyTools.DetectCostAnomalies` | Daily | warning if anomalies found, info if clean |
| `waste_scan` | `IdleResourceTools.FindIdleResources` | Weekly | warning if >5 idle resources, info otherwise |
| `maturity_score` | `ScoreTools` (7 Crawl dimensions) | Monthly | warning if any dimension regresses |
| `health_check` | `HealthTools.GetAzureServiceHealth` | Every 6 hours | critical if active incidents |
| `custom` | LLM invocation (future) | User-defined | User-defined |

For `custom` type schedules (arbitrary user prompts), defer LLM-based execution to a future iteration. The four built-in types cover the primary use cases without requiring LLM invocation in the background, which avoids the token cost and complexity.

## Files to Modify

| File | Change |
|------|--------|
| `src/Dashboard/AI/Tools/ScheduleTools.cs` | Add `UserId`, `ScheduleType`, `SubscriptionIds` fields; per-user storage; `UpdateLastRun()` and `GetDueSchedules()` methods |
| `src/Dashboard/Program.cs` | Register `ScheduleRunnerService` via `AddHostedService()` |

## New Files to Create

| File | Purpose |
|------|---------|
| `src/Dashboard/Services/ScheduleRunnerService.cs` | `BackgroundService` that evaluates and executes due schedules on a 5-minute timer |
| `src/Dashboard/Services/NotificationStore.cs` | Per-user notification persistence (file-based, scoped by userId) |
| `src/Dashboard/Endpoints/NotificationEndpoints.cs` | REST API for notification CRUD: list, unread count, mark read, dismiss |

## Frontend Changes

| File | Change |
|------|--------|
| `src/Dashboard/frontend/src/components/ChatView.vue` | Add bell icon with unread badge (60s poll); notification dropdown panel; click-to-view injects result into chat |

## Acceptance Criteria

- [ ] A new `ScheduleRunnerService` background service runs on a 5-minute timer and executes due schedules
- [ ] Schedules with `frequency: "daily"` execute once per 24-hour period; `"weekly"` once per 7 days; `"monthly"` once per calendar month
- [ ] Schedule execution uses managed identity (`DefaultAzureCredential`) for Azure ARM API access, not user-delegated tokens
- [ ] Built-in schedule types (`anomaly_check`, `waste_scan`, `health_check`) invoke tools directly without LLM, producing deterministic results
- [ ] `ReportSchedule` includes `UserId` field; schedules are stored per-user in `~/finops-agent-schedules/{userId}/`
- [ ] Schedule results are saved as notifications in `~/finops-agent-notifications/{userId}/` with a 30-day retention and 100-notification cap
- [ ] `GET /api/notifications` returns the current user's notifications (newest first)
- [ ] `GET /api/notifications/unread` returns the count of unread notifications
- [ ] `POST /api/notifications/{id}/read` marks a notification as read
- [ ] Frontend displays a bell icon with unread count badge, polled every 60 seconds
- [ ] Clicking a notification injects the full result into the chat view using the existing message rendering pipeline
- [ ] `ScheduleTools.SaveReportSchedule` accepts and stores the current user's deterministic userId
- [ ] A failing schedule does not block execution of other schedules
- [ ] Schedule execution is logged via OpenTelemetry with `finops.schedule.runs` counter (tags: schedule_id, frequency, status)
- [ ] Notifications are scoped by user — no user can read another user's notifications

## References

- [ASP.NET Core BackgroundService](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-9.0)
- [Azure Managed Identity with DefaultAzureCredential](https://learn.microsoft.com/en-us/dotnet/azure/sdk/authentication/credential-chains?tabs=dac)
- [Existing ScheduleTools.cs — GetAll() method](src/Dashboard/AI/Tools/ScheduleTools.cs) (designed for background consumer)
- [Existing UserStateJanitor.cs — BackgroundService pattern](src/Dashboard/Auth/UserStateJanitor.cs)

### Area

New AI tool (Azure / Graph / Log Analytics)

### Alternatives considered

### 1. Azure Functions timer trigger for scheduling

Deploy a separate Azure Function with a timer trigger (cron expression) that calls the FinOps agent's API. **Rejected** because it introduces a second deployment artifact, requires separate infrastructure (Function App + storage account, ~$5-10/mo), needs cross-service authentication, and duplicates the tool execution logic. The in-process `BackgroundService` is simpler, co-located with the tools, and has zero additional infrastructure cost.

### 2. Azure Logic Apps / Power Automate for orchestration

Use Logic Apps to schedule queries and route notifications. **Rejected** because it introduces a dependency on a separate Azure service ($15-50/mo depending on execution count), requires maintaining workflow definitions outside the codebase, and can't directly invoke the in-process C# tools. It adds operational complexity (monitoring two services) for a problem that an in-process timer solves natively.

### 3. Hangfire for background job scheduling

Use Hangfire (open-source .NET job scheduler) with a SQLite or Redis backend. **Rejected** because it introduces a significant dependency (Hangfire + dashboard UI + storage provider), is designed for job queuing at scale (thousands of jobs/sec) which is over-engineered for our use case (a few dozen schedules), and its dashboard adds a separate UI that would need to be secured. ASP.NET's built-in `BackgroundService` with `Task.Delay` is the right tool for a handful of timer-driven schedules.

### 4. Quartz.NET for cron scheduling

Use Quartz.NET for cron-expression-based scheduling. **Deferred** — Quartz.NET provides precise cron scheduling (e.g. "every Monday at 9:00 AM UTC"), job persistence, and misfire handling. However, it adds a dependency and requires a persistent job store (ADO.NET or MongoDB). The current requirement is for simple frequency-based schedules (daily/weekly/monthly), not precise cron timing. If users request specific time-of-day execution, Quartz.NET is the right upgrade path.

### 5. LLM-based execution for all scheduled tasks

Route all scheduled prompts through the Copilot SDK / LLM to interpret the prompt and choose which tools to call. **Rejected** for scheduled execution because: (1) LLM invocations cost $0.01–0.10 per call — running 20 daily schedules across 50 users costs $30-150/month in token spend alone; (2) LLM responses are non-deterministic — the same prompt may call different tools or produce different output formats; (3) creating a Copilot session requires a BYOK bearer token that currently refreshes per-user in HTTP context; (4) background LLM sessions have no user to display streaming output to. Direct tool invocation is deterministic, free, and faster. LLM-based execution is deferred to a `custom` schedule type in a future iteration.

### 6. WebSocket/SSE push for real-time notifications

Replace polling with a persistent WebSocket or SSE connection for instant notification delivery. **Deferred** — the existing chat SSE stream is per-request (opened on `POST /api/chat`, closed when the LLM finishes). A persistent notification channel would require keeping a long-lived connection per user, adding connection management complexity (heartbeats, reconnection, load balancer timeout configuration). Since background schedules run hourly/daily/weekly, 60-second polling is more than adequate and adds zero infrastructure complexity. WebSocket push can be added later if real-time alerts (sub-minute) become a requirement.

### 7. Email/Teams notification delivery

Send schedule results via email or Microsoft Teams webhook. **Deferred** — this is a high-value feature but orthogonal to the scheduling infrastructure. It requires either an SMTP relay / SendGrid account or a Teams incoming webhook URL, both of which need separate configuration and permissions. The in-app notification inbox is the right first delivery channel. Email/Teams can be added as additional delivery targets once the scheduler and notification store are operational.

### 8. Store notifications in Redis instead of files

Use the Redis cache (from the sessions issue) for notification storage. **Rejected** because notifications need to persist across Redis evictions (Redis is a cache, not a database), notification data can be large (full tool output JSON), and file-based storage in the App Service `/home` directory is durable, zero-cost, and requires no additional infrastructure. Redis is optimal for session tokens (small, transient); file storage is better for notification history (larger, longer-lived).