Graylog2 / Graylog2/graylog2-server

Content pack export produces dangling stream references when a referenced stream is unresolvable

Open
#26,584 0 comments 0 reactions 0 assignees View on GitHub
improvement triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

Creating a new revision of a content pack (re-exporting) can produce a dangling stream reference: a dashboard widget (or search type / query filter) references a stream by a content-pack id that has no matching `stream` or `stream_title` entity in the exported pack. Installing such a pack then fails with a missing stream exception.

This is the export-side root cause behind Graylog2/graylog-plugin-enterprise#12987. The import side is being hardened separately in Graylog2/graylog2-server#26581 (which makes such packs installable by dropping the unresolvable reference); this issue tracks stopping the broken pack from being produced in the first place.

## Confirmed root cause

The failure is the product of two decisions that combine badly during export:

1. **`StreamReferenceFacade.exportEntity`** loads the stream (`streamService.load()`) to emit the `stream_title` (`STREAM_REF_V1`) entity. If the load fails it returns `Optional.empty()`, so **no entity is emitted**.
2. The widget / search type / query filter that references the stream still serializes with the stream's content-pack UUID, because the stream is a node in the export dependency graph and therefore **received a UUID in `EntityDescriptorIds`** regardless of whether its entity was emitted.

Result: the exported pack contains a reference to a UUID that no entity in the pack defines → dangling reference → install fails.

`catch (Exception)` in `exportEntity` was broadened from `catch (NotFoundException)` in 7.0.1 (#24124, backport bd5885fb41), which widened *which* load failures silently drop the entity. Note, however, that the pre-7.0.1 code already dropped the entity for `NotFoundException` (a genuinely deleted stream), so the dangling-reference bug predates that change — #24124 only made it easier to hit.

### Reproduction (confirmed on Graylog 7.1.4, single node)

1. Create a stream titled `Evaluation: Self Monitoring Logs`.
2. Install the pack from `Graylog2/graylog-plugin-enterprise#12987` (rev 1) — succeeds.
3. Delete (or otherwise make unloadable) a stream that a dashboard widget references.
4. Re-export a new revision via `POST /api/system/catalog` with the installed native entity descriptors.
5. The exported rev 2 contains a widget/search-type reference to a UUID with no matching `stream`/`stream_title` entity.
6. Install rev 2 → **HTTP 500**, missing stream exception.

## Possible fixes

### A. Narrow the catch back to `NotFoundException`
- **Pro:** Small; restores fail-loud for non-not-found errors.
- **Con:** Insufficient — the `NotFoundException` path still produces a dangling reference; also risks re-introducing whatever #24124/#24167 fixed. Not viable alone.

### B. Fail export loudly when a referenced stream can't be resolved
Throw a clear `ContentPackException` naming the missing stream instead of emitting nothing.
- **Pro:** No silent data loss and no broken pack; the user learns at export time, the right moment to fix the source dashboard.
- **Con:** One stale reference blocks the entire revision, even for entities the user does not care about.

### C. Make the export self-consistent (recommended)
Ensure a stream that cannot be emitted does not receive a content-pack id, so the referencing widget/search-type/query naturally drops it (with a `WARN`). Cleanest seam: filter unresolvable stream descriptors at resolve time in `ContentPackEntityResolver` (it already has `StreamService` injected) before minting `EntityDescriptorIds`; also make the export-side mappers tolerant (`WidgetDTO.toContentPackEntity` already drops via `filter(Optional::isPresent)`; `SearchType.mappedStreams` currently throws and would need to drop instead).
- **Pro:** Produces an installable, self-consistent pack; revision creation is not blocked; symmetric with the import-side fix.
- **Con:** Silent semantic loss (widget loses a stream binding); touches the two-phase resolve/collect flow and requires aligning the widget vs. search-type mapper behavior.

### D. Import-side tolerance only (already in #26581)
- **Pro:** Makes any dangling pack installable regardless of source; zero export-side risk.
- **Con:** Broken packs are still produced; the binding is silently lost on install.

## Recommendation

Keep the import-side tolerance from #26581 as a safety net and add **C** so both export and import degrade gracefully and stay symmetric — this restores the invariant (the export never references an entity it did not emit) whose absence causes the bug. **B** is a reasonable alternative if the product view is that a content pack must be complete or refuse to export.

## References
- Import-side hardening: Graylog2/graylog2-server#26581
- Original report: Graylog2/graylog-plugin-enterprise#12987
- Related history: #24124 (broadened error handling), #24167 (stream `ObjectId` validation)

Contributor guide

Open the contributing guide

Research direction

Start with ContentPackEntityResolver’s resolve flow and its StreamService use, then read StreamReferenceFacade.exportEntity. Compare WidgetDTO.toContentPackEntity with SearchType.mappedStreams to see how unresolved streams are handled. Done means an exported pack contains no references to entities that were not emitted, while unresolved stream bindings are handled consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.