apache / apache/druid

New Unified Console Unusable for Large Clusters (1000+ datasources, 10k+ segments)

Open
#11,275 4 comments 0 reactions 0 assignees View on GitHub
Area - Web Console Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

After upgrading to 0.21.0 we noticed that the legacy UI (old-console) endpoints are not working anymore. our cluster is pretty big (thousands of data sources, a-lot of segments) - making the unified console in the router mode to not work.

### Affected Version
0.21.0 (and all versions when legacy UI isn't working anymore)
The Druid version where the problem was encountered.

### Description
When having:
- ~3500 data-sources
- ~20,000 segments

Router mode:
- Datasources page isn't working in router mode - the SQL query endpoint hangs
- Segments page isn't working in router mode - SQL hangs

Coordinator mode:
- Datasources page working well in coordinator mode- but without the segment timeline view
- Segments page crashing in coordinator mode - making thousands of requests - one per datasource - crashing chrome with insufficient resources

More:
- no way to filter prior query in both segments and datasources page
- no way to save a preset of filters with the responsive UI
- Ingestion page seems to work well even under high pressure of tasks (20-30k daily tasks)

All of the above worked well in the old-console - which we didn't find a way to enable in 0.21.0 - we saw a commit completely remove the binding so I believe there's no way to use it anymore.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the unified console's Datasources and Segments pages in router and coordinator modes with the reported scale of about 3,500 data sources and 20,000 segments. Inspect the SQL query endpoints and the per-datasource requests, then compare their behavior with the unavailable old-console endpoints. Done means both pages remain usable at large-cluster scale without hanging, crashing the browser, or issuing excessive requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
backend, frontend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.