Indicate Partial Response in query response when servers have not loaded all segments
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
We have observed the following behavior:
1. We have an OFFLINE table
2. We load 1000 segments via the SegmentGenerationAndPush minion-based task; they are about 500MB each
3. The minions finish all the subtasks, and the `SegmentGenerationAndPushTask` is complete.
4. We run a `select COUNT(*) from table;` query, and continually get larger and larger numbers, as we start to hit more and more segments (as seen in the `numSegmentsQueried`; that is, we hit fewer than the idealState of segments) and so are getting partial results until all segments are fully available (or in the routing table, not sure exact terminology here). We expect this to only return correct results for `numSegmentsQueried: 1000`, since the broker can't optimize hitting fewer than `idealstate` segments, for this sort of query (aggregation across all segments with no time/segment partition-based selection) as we understand it.
5. There is no indication in the query response that we are querying fewer than the idealState of segments; this is unlike if a server has just gone down, in which we get an `exception:` in the query response.
This also appears to happen if a server comes back online, and has segments assigned but not fully queryable. (That is, a server has `BAD` segments but we get no indication that are not querying these in the query response)
We'd either like something in the query response indicating that fewer-than-ideal segments are available, or some flag to fail queries in this case (I guess the former is more flexible for clients to decide how to handle).
This may be a dupe of https://github.com/apache/pinot/issues/7130, as perhaps this is the same case as (2) there ("Routing table doesn't contain full segments, e.g. all replicas are down for a segment in idealStates.")/
Contributor guide
Research direction
Start by tracing the query response behavior around numSegmentsQueried and the existing exception reported when servers are unavailable. Done means a query that reaches fewer than the idealState segments clearly indicates a partial response, with behavior that distinguishes this from a fully served query.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100