apache / apache/skywalking

[Chore] Keep Stream filtering and projection on batches until output

Open
#14,067 0 comments 0 reactions 0 assignees View on GitHub
database
Dominant language
Java
Stars
25k
Forks
6.6k
Avg merge
10h 5m
Merged PRs (30d)
16

Description

## Summary

Stop constructing protobuf `Element` objects merely to filter rows or remove internally projected tags. This follows the legacy-engine removal child #14066 under #13998.

## Size audit and dependencies

Classification: **tracking parent**, not an executable leaf or automated implementation task.
Boundary: Stream batch filtering and client projection through `banyand/query/processor.go`, with eligible frame egress and required protobuf egress.
It spans filter placement, projection, and response-path activation. Split into executable tasks against the merged implementation rather than assigning the entire workstream as one PR.
Blocked by: #14066, implemented by BanyanDB PR apache/skywalking-banyandb#1326 (still open when this issue was drafted).
Prerequisite already merged: BanyanDB PR apache/skywalking-banyandb#1331 provides pre-merge filtering for index-order queries.

## Requirements

R1. Reuse apache/skywalking-banyandb#1331's pre-merge index-order filtering and remove the redundant element-level recheck. Preserve filter-first duplicate winners and the bounded merge.
R2. Apply timestamp-order filtering to batches at its existing position **after the cap**; do not change its existing under-fill behavior.
R3. Strip criteria-only and hidden ordering tags through batch projection. Retain internal ordering keys until ordering is finished, without exposing hidden tags in the client's projection.
R4. Enable frame output for eligible distributed queries. Preserve required protobuf responses, including traced responses and final client output.

## Concrete oracle

Input:

```text
ID=A, order=1, state=closed, service=old
ID=A, order=2, state=open, service=new
ID=B, order=3, state=open, service=other
```

With criteria `state=open`, ascending index order, and projection `service`, return A/new then B/other. Neither `state` nor the ordering tag may appear in client tag families. Offset 1, limit 1 returns B/other. Expected values are specified here, not derived from a legacy engine.

## Completion criteria and RED-test direction

- [ ] Real processor tests prove eligible filtered queries emit frames. Today their filter gate selects protobuf output, so this is an observable failing path assertion, not only output parity.
- [ ] Exact selected values, duplicate winners, hidden-tag removal, null behavior, and offset/limit are tested across batches.
- [ ] Timestamp-order filtering remains behind the cap, and traced responses retain tracing information.
- [ ] Each implementation leaf activates a production caller and includes its own failing test; batch ownership is correct on success, cancellation, and errors.

```sh
go test ./pkg/query/vectorized/stream/... ./pkg/query/logical/stream ./banyand/query
go test ./test/integration/standalone/query/... ./test/integration/distributed/query/...
```

## Out of scope

Reimplementing apache/skywalking-banyandb#1331; changing timestamp under-fill semantics; local cross-group or distributed result merging (a separate workstream).

Parent: #13998.

## Compatibility and review policy

- Preserve existing on-disk and wire formats and mixed-version compatibility. Any necessary protocol change requires a separately reviewed compatibility contract.
- Use explicit, hand-calculated expected results; do not depend on the removed row engine as the test oracle.
- Human review enforces the no-new-row-path policy. A row-call-site lint guard is not required.

Contributor guide

Open the contributing guide

Research direction

Start with banyand/query/processor.go and the prerequisite changes in apache/skywalking-banyandb#1331, then run the listed Go and integration test commands. This issue is a tracking parent, so split the work into executable leaf tasks before implementation. Done means each leaf has a production caller, failing processor or integration coverage, and preserves the listed filtering, projection, response, and tracing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.