airbytehq / airbytehq/PyAirbyte
Reevaluate MCP telemetry identity: Segment write key and user/analytics ID
- Dominant language
- Python
- Stars
- 344
- Forks
- 77
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 35
Description
## Context
[#1124](https://github.com/airbytehq/PyAirbyte/pull/1124) wires the shared `fastmcp-extensions` tool-call telemetry into PyAirbyte's MCP server, so every tool call (stdio and hosted HTTP) now emits an `mcp_tool_call` Segment event.
Two identity/attribution decisions were made expediently in that PR and deserve a real design pass before we build reporting on top of them.
## 1. Segment write key
Today the MCP tool-call stream defaults to PyAirbyte's existing app tracking key (`PYAIRBYTE_APP_TRACKING_KEY`), overridable per-deployment via the `AIRBYTE_MCP_SEGMENT_WRITE_KEY` env var, which no deployment currently sets. Consequences:
- Hosted Cloud MCP, local stdio MCP, and plain PyAirbyte library usage all share one Segment source, so they can only be separated downstream by event name and the `is_hosted_mcp` property — not by source.
- Volume/retention/sampling policy cannot be tuned for MCP traffic independently of library traffic.
- Rotating or revoking the key affects both streams.
Options to evaluate: keep the shared key; a single MCP-specific key set on all MCP deployments; or a per-deployment key (Cloud MCP support bot vs. local vs. other hosted MCP servers), which is already possible by setting `AIRBYTE_MCP_SEGMENT_WRITE_KEY` and requires no code change here.
## 2. `user_id` / analytics ID
The shared middleware sends a fixed `user_id` (`airbyte-mcp` for PyAirbyte's server) and no `anonymousId`. That is deliberately privacy-safe, but it means:
- We can count tool calls, but not unique callers, workspaces, or organizations.
- Hosted Cloud MCP serves the Airbyte Cloud support bot and many Devin sessions from one container, so even the pre-existing PyAirbyte `anonymous_id` (derived from `~/.airbyte/analytics.yml` on the local filesystem) would be per-container, not per-caller — it is not a usable substitute.
- The regular PyAirbyte `install`/`sync` streams *do* carry `anonymous_id`, so MCP events are a different shape from everything else in the same dataset.
Candidates worth weighing against privacy/PII constraints:
- Keep the fixed label (status quo): counts only.
- Local stdio: reuse the existing PyAirbyte `anonymous_id` so local MCP matches the rest of PyAirbyte telemetry.
- Hosted: derive a stable, non-PII caller identifier from the authenticated request (e.g. a hash of workspace/organization ID, or of the client ID) and send it as `anonymousId` or a property — never a raw email or user ID.
- Distinguish the calling client (support bot vs. Devin vs. human IDE) via an explicit client-supplied property rather than inferring it.
## Why it matters
[airbyte-warehouse#3472](https://github.com/airbytehq/airbyte-warehouse/pull/3472) models this stream and [internal.airbyte.ai#128](https://github.com/airbytehq/internal.airbyte.ai/pull/128) will report on it. Any "unique users of the MCP" metric is impossible until this is settled, and changing identity semantics after the fact makes historical data non-comparable — so it is cheaper to decide now, while the stream is new.
## Scope
Decision + follow-up implementation; no change required to unblock #1124.
Contributor guide
Assessment
This issue has not been assessed yet.