Dashboard can become unresponsive after long-running continuous log/trace ingestion
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
## Summary
After leaving an Aspire dashboard running for a long-lived local AppHost with continuous console logs, browser logs, and OTLP trace ingestion, the dashboard became partially unresponsive. The dashboard process had been running for about **34.9 hours** when investigated.
The most visible symptom was that sidebar navigation updated the browser URL, but the rendered Blazor UI stayed on the previous logs page. Direct URL loads still worked for some pages, so this looked like a wedged Blazor Server/dashboard state issue rather than a full process crash.
## What happened
- The Console and Structured Logs pages had accumulated a large amount of data.
- Clicking sidebar items such as Traces, Metrics, or Resources changed `location.pathname`, but the page title, H1, and body stayed on the previous Structured Logs page.
- Direct navigation to some URLs still rendered correctly.
- One direct navigation to `/traces` rendered a blank page during the investigation.
- Browser console did not show a useful JavaScript exception; the Blazor WebSocket was connected.
- During failing navigation, the browser sent Blazor event frames, but the expected render response did not arrive.
## CLI diagnostics at scale
I tried the CLI diagnostics commands because they are the tools I would expect to use when the dashboard UI itself is slow or partially wedged.
What worked well:
- `aspire otel logs --limit 20 --format Json --non-interactive` completed in about **1.1s**.
- `aspire otel logs --limit 200 --format Json --non-interactive` completed in about **1.3s**.
- The structured log IDs were already around **490k**, so this was useful even with a substantial amount of retained log data.
- `aspire logs --include-hidden --tail 20 --format Json --non-interactive` completed in about **1.2s** and surfaced dashboard-internal logs, which was useful for identifying server-side/circuit symptoms.
What struggled:
- `aspire logs --tail 20 --format Json --non-interactive` across all visible resources did not complete within **60s**; it stayed at `Getting logs...`.
- `aspire logs --include-hidden --tail 20 --format Json --non-interactive` across all resources also did not complete within **60s**.
- `aspire logs --tail 20 --format Json --non-interactive` for a high-volume browser/console-log resource did not complete within **60s**.
- `aspire otel traces --limit 20 --format Json --non-interactive` did not complete within **60s**.
This made `aspire otel logs` very useful as a fallback when the dashboard UI was sluggish, but the console-log path (`aspire logs`) appeared to hit the same scale problem as the UI when querying all resources or a high-volume console log resource. The trace query path also appeared affected.
It would be great if the CLI paths stayed responsive at this scale, especially with small limits/tails, because they are the primary escape hatch when the dashboard UI is no longer reliable.
## Profiling notes
Using `aspire resources --include-hidden --format Json`, the hidden `aspire-dashboard` executable resource was identified and profiled locally.
At the time of profiling, the dashboard process was not CPU-bound, but it was very large:
- Private memory: approximately **2.2 GB** before collection, approximately **1.86 GB** after a GC dump collection
- Working set: approximately **1.7 GB** before collection, approximately **1.43 GB** after collection
- Threads: approximately **650-690**
- Managed heap from `dotnet-gcdump`: approximately **956 MB**, **7.8M objects**
Notable managed heap entries included:
| Type | Approx retained/count |
|---|---:|
| `System.String` | ~254 MB / ~4.0M objects |
| `Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]` | ~111 MB |
| `System.Byte[]` | ~105 MB |
| `Aspire.Dashboard.Otlp.Model.OtlpSpan` | ~20 MB / ~139k objects |
| OTLP `AnyValue` / `KeyValue` | ~18 MB each / ~481k objects each |
| `OpenTelemetry.Proto.Trace.V1.Span` | ~7 MB / ~54k objects |
| `Aspire.Shared.ConsoleLogs.LogEntry` | ~2.8 MB / ~40k objects |
A mini dump stack sample showed multiple requests in the OTLP trace ingestion path waiting on the telemetry repository write lock:
```text
System.Threading.ReaderWriterLockSlim.TryEnterWriteLockCore(...)
Aspire.Dashboard.Otlp.Storage.TelemetryRepository.AddTracesCore()
Aspire.Dashboard.Otlp.Storage.TelemetryRepository.AddTraces()
Aspire.Dashboard.Otlp.OtlpTraceService.Export()
Aspire.Dashboard.Otlp.Http.OtlpHttpEndpointsBuilder...
```
This suggests the dashboard may be struggling under sustained trace/log ingestion, and the large logs page render tree may also contribute to Blazor Server circuit responsiveness problems.
## Environment
- Aspire SDK: `13.3.0-preview.1.26227.17`
- Channel: daily
- OS: Windows
- .NET SDK observed locally: `10.0.107`
- Dashboard uptime at the time of investigation: about **34.9 hours**
- Scenario: local AppHost with continuous console logs, browser logs, and OTLP trace ingestion over a long-running session
## Desired behavior
The dashboard should continue to navigate between pages reliably after long-running log/trace ingestion, or it should bound/evict enough telemetry/render state that log-heavy pages do not wedge the Blazor Server circuit or log query path.
The CLI diagnostics paths should also stay responsive with small `--tail` / `--limit` values even when the dashboard has accumulated a large amount of telemetry, because these commands are the most useful fallback when the UI is degraded.
## Notes
I am not attaching dumps publicly because they may contain local telemetry/log data, but I can provide more sanitized details if helpful.
Contributor guide
Assessment
This issue has not been assessed yet.