apache / apache/pinot

Indicate Partial Response in query response when servers have not loaded all segments

Open
#10,076 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

We have observed the following behavior:
1. We have an OFFLINE table
2. We load 1000 segments via the SegmentGenerationAndPush minion-based task; they are about 500MB each
3. The minions finish all the subtasks, and the `SegmentGenerationAndPushTask` is complete.
4. We run a `select COUNT(*) from table;` query, and continually get larger and larger numbers, as we start to hit more and more segments (as seen in the `numSegmentsQueried`; that is, we hit fewer than the idealState of segments) and so are getting partial results until all segments are fully available (or in the routing table, not sure exact terminology here). We expect this to only return correct results for `numSegmentsQueried: 1000`, since the broker can't optimize hitting fewer than `idealstate` segments, for this sort of query (aggregation across all segments with no time/segment partition-based selection) as we understand it.
5. There is no indication in the query response that we are querying fewer than the idealState of segments; this is unlike if a server has just gone down, in which we get an `exception:` in the query response.

This also appears to happen if a server comes back online, and has segments assigned but not fully queryable. (That is, a server has `BAD` segments but we get no indication that are not querying these in the query response)

We'd either like something in the query response indicating that fewer-than-ideal segments are available, or some flag to fail queries in this case (I guess the former is more flexible for clients to decide how to handle).

This may be a dupe of https://github.com/apache/pinot/issues/7130, as perhaps this is the same case as (2) there ("Routing table doesn't contain full segments, e.g. all replicas are down for a segment in idealStates.")/

Contributor guide

Open the contributing guide

Research direction

Start by tracing the query response behavior around numSegmentsQueried and the existing exception reported when servers are unavailable. Done means a query that reaches fewer than the idealState segments clearly indicates a partial response, with behavior that distinguishes this from a fully served query.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.