apache / apache/pulsar

[Question][scalable-topics] What is the full parent-before-child delivery barrier?

Open
#26,274 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

## Context

PIP-460 states that all messages in a parent segment precede messages in its children and that consumers traverse the DAG in that order across split and merge.

In `v5.0.0-M1` and current `master`, `SubscriptionCoordinator.isAssignable()` appears narrower:

- every sealed segment is immediately assignable;
- only an active child checks whether all direct parents are in `drainedSegmentIds`.

This leaves a deep DAG case where a sealed intermediate child can be delivered alongside its undrained sealed ancestor. Existing multi-generation replay tests check completeness, not parent-before-child order. Merge code checks all direct parents by inspection, but I could not find a StreamConsumer test with unread backlog on every merge parent. `drainedSegmentIds` also appears monotonic even though seek can rewind sealed cursors.

## Questions

1. Does the parent-before-child guarantee cover sealed intermediate segments in a deep DAG?
2. Is an active merge child guaranteed to remain unavailable until every parent cursor is drained?
3. Should seek or cursor rewind invalidate and recompute `drainedSegmentIds` and descendant eligibility?
4. Is a non-Java client expected to implement a full local DAG barrier, or rely exclusively on controller assignments?
5. Does "processed" mean the application must acknowledge only after processing, since the controller can observe cursor backlog but not application completion?

It would help to have authoritative tests for deep split ancestry, merge with unread data on all parents, seek after a parent was marked drained, and controller failover during draining.

References: PIP-460, PIP-468, `SubscriptionCoordinator.isAssignable`, `V5StreamConsumerDagReplayTest`, and the scalable seek implementation.

Contributor guide

Open the contributing guide

Research direction

Start with PIP-460 and PIP-468, then inspect SubscriptionCoordinator.isAssignable and the scalable seek implementation. Run or extend V5StreamConsumerDagReplayTest to cover deep split ancestry, merges with unread data on every parent, seek after draining, and controller failover. Done means the parent-before-child and rewind semantics are answered by authoritative tests or documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.