api7 / api7/aisix

OtlpHttpFanOut::gc has no callers, so pipelines for deleted exporters are never stopped

Open
#954 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
157
Forks
32
Avg merge
1h 25m
Merged PRs (30d)
145

Description

OtlpHttpFanOut::gc (crates/aisix-obs/src/otlp_http_sink.rs:263)
documents itself as "Called periodically by the server to GC pipelines for
deleted / disabled exporters", but git grep '\.gc(' returns no callers
anywhere in the tree. Nothing ever reaps them.

Consequence: when an operator deletes or disables an observability
exporter, the fan-out stops handing it events (the exporter is gone from
the snapshot), but its pipeline — task, client, queue — stays alive for the
life of the process. A deployment that churns exporters accumulates them.

Found while auditing api7/aisix#953, which tightened the adjacent
behaviour: the fan-out now re-reads the live snapshot rather than the
request's frozen one, so a deleted exporter stops receiving events
immediately. That closes the delivery side; this issue is the resource side,
and it predates that PR — gc has never been wired.

Worth deciding between two shapes rather than just adding a caller:

  • call gc from the existing snapshot-apply path, so reaping is driven by
    the config change that caused it, or
  • call it from a periodic task, which is what the doc comment assumes.

The first is more precise and has no idle cost; the second is more forgiving
if a pipeline can outlive its exporter for reasons other than deletion.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in crates/aisix-obs/src/otlp_http_sink.rs:263 and inspect OtlpHttpFanOut::gc, then trace the existing snapshot-apply path. Use git grep '.gc(' to confirm the missing caller and compare whether snapshot-driven or periodic invocation fits the surrounding lifecycle. Done means deleted or disabled exporter pipelines are reaped without accumulating tasks, clients, or queues.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.