apache / apache/druid

Better handling of different servers having different versions of broadcast table segments

Open
#10,444 0 comments 0 reactions 0 assignees View on GitHub
Area - Querying Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description
Broadcast data sources can potentially have different segment versions on different servers. It can result in some inconsistencies
- If a query involving broadcast data source is evaluated on different servers (such as join), the end result might be incorrect since different servers could have different versions of data.
- Broker may cache the result of join queries on its end. During the query execution, historicals may have an older version of data. However, once the broker caches it, the historicals will not be re-queried even if they update their data with a newer version of data. Thus, brokers will be left with stale query results. The scenario is described here - https://github.com/apache/druid/pull/10366/files#r494684538

Contributor guide

Open the contributing guide

Research direction

Start by reading the broadcast data source, broker, historical, and join-query scenarios described here, then review the linked discussion in PR #10366. Define how differing segment versions should be handled across servers and broker caching, with completion requiring consistent query results and no stale cached joins after historicals update.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.