labring / labring/FastGPT

feat: Integrate LLM Tracing platforms

Open
#6,714 0 comments 0 reactions 1 assignee Claimed by @ctlaltlaltc View on GitHub
feature
Dominant language
TypeScript
Stars
29.6k
Forks
7.3k
Avg merge
1d 7h
Merged PRs (30d)
152

Description

## Overview

Integrate external LLM Tracing/observability platforms into FastGPT to track all LLM calls — inputs, outputs, token usage, latency, and errors — for debugging, cost analysis, and quality evaluation.

**Initial implementation:** Langfuse
**Architecture:** Extensible provider abstraction to support LangSmith, Opik, LangWatch, Arize Phoenix, LunaryAI in future

---

## Use Cases

- Debug why an agent gave an unexpected answer by inspecting the full message chain
- Analyze token costs per app, team, or model
- Measure response latency and Time-To-First-Token across different models
- Audit all LLM calls for compliance or quality review
- Compare prompt versions using Langfuse's built-in scoring and evaluation tools

---

## User Stories

**As a platform operator**, I want to configure a global LLM Tracing provider via environment variables so that all LLM calls are tracked by default.

**As a team admin**, I want to configure one or more Tracing providers for my team so that all apps under my team send data to our observability platform.

**As an app owner**, I want to disable Tracing for a specific app even if the team has it enabled, so sensitive apps are excluded.

---

## Configuration Priority

```
App (switch only) > Team (credentials) > System (env vars)
```

- **System level** — env vars, holds credentials, applies to all LLM calls
- **Team level** — MongoDB, holds credentials, supports multiple providers, applies to all apps in team
- **App level** — MongoDB, switch only (`enabled: true/false`), overrides team config

Team supports **multiple providers simultaneously** — data is fanned out to all enabled providers in parallel.

---

## Architecture Integration

**Single integration point:** `createLLMResponse()` in `packages/service/core/ai/llm/request.ts`

All LLM calls (chat, agent loops, classify, extract, etc.) flow through this function. Tracing is injected here — async, non-blocking, failure-tolerant.

```
Request entry (v1/v2/completions)
→ generate traceId
→ create Trace + Span in Langfuse
→ pass tracingContext through DispatchFlowProps
→ createLLMResponse() reports each Generation
```

**Trace hierarchy in Langfuse:**
```
Trace (one per request)
└── Span: workflow-run
├── Generation: llm-chat
├── Generation: llm-agent-round-1
└── Generation: llm-extract
```

**New files:**
```
packages/global/core/ai/tracing/type.ts # types
packages/service/core/ai/tracing/
├── config.ts # priority resolution
├── manager.ts # multi-provider fan-out
└── providers/
├── base.ts # abstract provider
└── langfuse.ts # Langfuse implementation
```

**Schema changes:** Add `llmTracing.providers[]` to `TeamSchema`, add `llmTracing.enabled` to `AppSchema`.

---

## Scope

- [ ] Langfuse provider (SDK integration, Trace/Span/Generation hierarchy, streaming TTFT)
- [ ] Multi-provider manager (parallel fan-out via `Promise.allSettled`)
- [ ] Config resolution with 3-level priority
- [ ] Team API: CRUD providers + test connection
- [ ] App API: toggle enabled/disabled
- [ ] Team settings UI: add/remove/edit providers
- [ ] App settings UI: enable/disable toggle
- [ ] Unit tests: provider, config priority, multi-provider failure isolation

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.