grafana / grafana/fullstack-o11y-examples

All 5 language stacks hardcode identical service.instance.id (+ namespace + job), cross-contaminating Knowledge Graph entities

Open
#69 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2
Forks
0
Avg merge
5d 2h
Merged PRs (30d)
41

Description

# All 5 language stacks hardcode identical `service.instance.id` (+ namespace + job), cross-contaminating Knowledge Graph entities

**Repo:** `grafana/fullstack-o11y-examples` · **Area:** docker-compose / OTel resource attributes · **Type:** bug

## Summary

All five language stacks (Java, Go, Node.js, Python, Ruby) emit OTel telemetry with the **exact same** `service.namespace=bookstore` + `service.instance.id=bookstore-{checkout,products,shipping}` + `job` values. Nothing in the resource attributes distinguishes which language stack produced a given data point.

When more than one stack is run against the **same** Grafana Cloud target — even sequentially, with `docker compose down -v` between runs — they collide on the same Knowledge Graph entity identity. Because the KG keys `Service` / `ServiceInstance` entities on `job` + `instance`, the entity becomes a cross-stack property union and gets mislabeled (e.g. JVM runtime fields while a non-JVM stack is the only one running).

> The downstream "stale properties never reset" behavior is a platform-side model-builder concern tracked separately in the Knowledge Graph / Asserts backend (**grafana/asserts-adi#3556**). **This** issue is the config trigger in this repo, which is fixable here.

## Root cause

```
$ grep -rn "service.instance.id=bookstore" */docker-compose/docker-compose.yml
java/docker-compose/docker-compose.yml: service.instance.id=bookstore-checkout, ...
python/docker-compose/docker-compose.yml: service.instance.id=bookstore-checkout, ...
go/docker-compose/docker-compose.yml: service.instance.id=bookstore-checkout, ...
nodejs/docker-compose/docker-compose.yml: service.instance.id=bookstore-checkout, ...
ruby/docker-compose/docker-compose.yml: service.instance.id=bookstore-checkout, ...
```

Same pattern for `products` and `shipping`. The k8s manifests share the identity too.

## Impact

- **Entity contamination:** running any second stack against a target that a first stack already touched permanently mislabels the shared entity's properties (`runtime`, `process_runtime_name`, `telemetry_sdk_language`, …). Observed live against a shared Grafana Cloud KG target: with only the **Node.js** stack running and freshly traffic'd, all three app-service entities still reported `runtime: jvm` / `telemetry_sdk_language: python`.
- **Breaks clean per-stack validation:** you cannot get trustworthy per-stack *property* evidence by testing one stack at a time against a shared account — whichever stack ran first lingers regardless of teardown. (Edge presence/absence — CALLS, Service→Database — is still trustworthy, since edges come from concurrently-active spans, not accumulated properties.)

## Proposed fix

Pick one:

1. **Distinct identity per stack (preferred):** suffix `service.instance.id` with the language in all five `*/docker-compose/docker-compose.yml` and the k8s manifests, e.g. `bookstore-checkout-java`, `bookstore-checkout-node`, etc. (or drop the hardcoded value and let the SDK generate one).
2. **Document the constraint:** if the stacks are intended to demonstrate the *same* logical service in different languages, state clearly that they must run against **separate** Grafana Cloud targets/environments — never the same one simultaneously or in sequence without an env/instance change.

Recommend (1): it keeps the demo self-contained and makes cross-language comparisons possible on one target.

## Evidence / reference

Full validation write-up with live `gcx kg` queries: `docs/kg-validation.md` (Finding B/D and the per-stack evidence sections). This issue corresponds to that doc's **Recommended next step #3**.

Companion platform discussion (stale-property lifecycle), with the full reproduction and evidence: grafana/asserts-adi#3556 — https://github.com/grafana/asserts-adi/issues/3556#issuecomment-5318954876

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the five */docker-compose/docker-compose.yml files identified by the grep example, then inspect the corresponding k8s manifests and docs/kg-validation.md. Confirm the repeated service.instance.id, service.namespace, and job values, then verify that each language stack has distinct service identities across both deployment formats and that no identical hardcoded identity remains.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, go, java, kubernetes, node.js, python, ruby
Domain
devops, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.