New Unified Console Unusable for Large Clusters (1000+ datasources, 10k+ segments)
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
After upgrading to 0.21.0 we noticed that the legacy UI (old-console) endpoints are not working anymore. our cluster is pretty big (thousands of data sources, a-lot of segments) - making the unified console in the router mode to not work.
### Affected Version
0.21.0 (and all versions when legacy UI isn't working anymore)
The Druid version where the problem was encountered.
### Description
When having:
- ~3500 data-sources
- ~20,000 segments
Router mode:
- Datasources page isn't working in router mode - the SQL query endpoint hangs
- Segments page isn't working in router mode - SQL hangs
Coordinator mode:
- Datasources page working well in coordinator mode- but without the segment timeline view
- Segments page crashing in coordinator mode - making thousands of requests - one per datasource - crashing chrome with insufficient resources
More:
- no way to filter prior query in both segments and datasources page
- no way to save a preset of filters with the responsive UI
- Ingestion page seems to work well even under high pressure of tasks (20-30k daily tasks)
All of the above worked well in the old-console - which we didn't find a way to enable in 0.21.0 - we saw a commit completely remove the binding so I believe there's no way to use it anymore.
Contributor guide
Research direction
Start by reproducing the unified console's Datasources and Segments pages in router and coordinator modes with the reported scale of about 3,500 data sources and 20,000 segments. Inspect the SQL query endpoints and the per-datasource requests, then compare their behavior with the unavailable old-console endpoints. Done means both pages remain usable at large-cluster scale without hanging, crashing the browser, or issuing excessive requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- backend, frontend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100