broker metadata refresh exception, This leads to an infinite loop of retries, and the broker is unable to retrieve the corresponding metadata.

The issue occurred in version Druid 28.0.1

broker keeps reporting errors when loading and refreshing segments. After calling the indexer and receiving an error, for some reason, the indexer fails to query the datasource. This causes the broker's segment cache to refresh and retry indefinitely, resulting in abnormal metadata cache.

I believe this is an issue of robustness that needs to be addressed.

Historical issue : https://github.com/apache/druid/issues/16936

When `indexer.getBundle()` is null, an IAE (IllegalArgumentException) is thrown. After the broker captures this exception, it will be added back to the refresh queue, leading to repeated retries and ultimately causing an infinite loop.

@VisibleForTesting
DatasourceBundle getBundle(final Query query)
{
final DataSourceAnalysis analysis = query.getDataSource().getAnalysis();

final TableDataSource table =
analysis.getBaseTableDataSource()
.orElseThrow(() -> new ISE("Cannot handle datasource: %s", query.getDataSource()));

final DatasourceBundle bundle;

synchronized (this) {
bundle = datasourceBundles.get(table.getName());
}

if (bundle == null) {
throw new IAE("Could not find segment walker for datasource [%s]", table.getName());
}

return bundle;
}

The following code will result in an infinite loop.

refresh(segmentsToRefresh, dataSourcesToRebuild);

initialized.countDown();
}
catch (InterruptedException e) {
// Fall through.
throw e;
}
catch (Exception e) {
log.warn(e, "Metadata refresh failed, trying again soon.");

synchronized (lock) {
// Add our segments and dataSources back to their refresh and rebuild lists.
segmentsNeedingRefresh.addAll(segmentsToRefresh);
dataSourcesNeedingRebuild.addAll(dataSourcesToRebuild);
lastFailure = System.currentTimeMillis();
}
}

The error stack is as follows:

broker.log
[DruidSchema-Cache-0][ROOT][org.apache.druid.client.JsonParserIterator] Query [12907933-7fdd-4c09-889c-f0c774a0ec91] to host [xx.xx.xx.xx:26204] interrupted
org.apache.druid.query.QueryException: Could not find segment walker for datasource [tbl_data_res_a]
at sun.reflect.GeneratedConstructorAccessor411.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_452]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_452]
at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:124) ~[jackson-databind-2.12.7.jar:2.12.7]
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:291) ~[jackson-databind-2.12.7.jar:2.12.7]
at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:288) ~[jackson-databind-2.12.7.jar:2.12.7]
at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:202) ~[jackson-databind-2.12.7.jar:2.12.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:454) ~[jackson-databind-2.12.7.jar:2.12.7]
at com.fasterxml.jackson.databind.deser.std.ThrowableDeserializer.deserializeFromObject(ThrowableDeserializer.java:65) ~[jackson-databind-2.12.7.jar:2.12.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195) ~[jackson-databind-2.12.7.jar:2.12.7]
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322) ~[jackson-databind-2.12.7.jar:2.12.7]
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4569) ~[jackson-databind-2.12.7.jar:2.12.7]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2798) ~[jackson-databind-2.12.7.jar:2.12.7]
at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:186) ~[druid-server-28.0.1-htrunk50.jar:?]
at org.apache.druid.client.JsonParserIterator.hasNext(JsonParserIterator.java:92) ~[druid-server-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:70) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.MergeSequence.lambda$toYielder$0(MergeSequence.java:66) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:63) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.MergeSequence.lambda$toYielder$0(MergeSequence.java:66) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:63) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:133) ~[druid-server-28.0.1-htrunk50.jar:?]
at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:77) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.Yielders.each(Yielders.java:32) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.sql.calcite.schema.SegmentMetadataCache.refreshSegmentsForDataSource(SegmentMetadataCache.java:796) ~[druid-sql-28.0.1-htrunk50.jar:?]
at org.apache.druid.sql.calcite.schema.SegmentMetadataCache.refreshSegments(SegmentMetadataCache.java:744) ~[druid-sql-28.0.1-htrunk50.jar:?]
at org.apache.druid.sql.calcite.schema.SegmentMetadataCache.refresh(SegmentMetadataCache.java:467) ~[druid-sql-28.0.1-htrunk50.jar:?]
at org.apache.druid.sql.calcite.schema.SegmentMetadataCache.lambda$startCacheExec$1(SegmentMetadataCache.java:409) ~[druid-sql-28.0.1-htrunk50.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_452]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_452]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_452]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_452]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_452]
2026-05-06 07:02:37,161 WARN [DruidSchema-Cache-0][ROOT][org.apache.druid.server.QueryLifecycle] Exception while processing queryId [12907933-7fdd-4c09-889c-f0c774a0ec91]
org.apache.druid.query.QueryInterruptedException: Could not find segment walker for datasource [tbl_data_res_a]
at org.apache.druid.client.JsonParserIterator.convertException(JsonParserIterator.java:276) ~[druid-server-28.0.1-htrunk50.jar:?]
at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:186) ~[druid-server-28.0.1-htrunk50.jar:?]
at org.apache.druid.client.JsonParserIterator.hasNext(JsonParserIterator.java:92) ~[druid-server-28.0.1-htrunk50.jar:?]
at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:70) ~[druid-processing-28.0.1-htrunk50.jar:?]

indexer.log
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.servlets.QoSFilter.doFilter(QoSFilter.java:202) ~[jetty-servlets-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_452]
Caused by: org.apache.druid.query.QueryInterruptedException: Could not find segment walker for datasource [tbl_data_res_a]
... 86 more
Caused by: org.apache.druid.java.util.common.IAE: Could not find segment walker for datasource [tbl_data_res_a]
at org.apache.druid.segment.realtime.appenderator.UnifiedIndexerAppenderatorsManager.getBundle(UnifiedIndexerAppenderatorsManager.java:376) ~[druid-server-28.0.1-htrunk50.jar:?]
at org.apache.druid.segment.realtime.appenderator.UnifiedIndexerAppenderatorsManager.getQueryRunnerForSegments(UnifiedIndexerAppenderatorsManager.java:357) ~[druid-server-28.0.1-htrunk50.jar:?]
at org.apache.druid.indexing.overlord.ThreadingTaskRunner.getQueryRunnerForSegments(ThreadingTaskRunner.java:550) ~[druid-indexing-service-28.0.1-htrunk50.jar:?]
at org.apache.druid.query.spec.MultipleSpecificSegmentSpec.lookup(MultipleSpecificSegmentSpec.java:77) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.query.BaseQuery.getRunner(BaseQuery.java:119) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.query.QueryPlus.run(QueryPlus.java:149) ~[druid-processing-28.0.1-htrunk50.jar:?]
at org.apache.druid.server.QueryLifecycle.execute(QueryLifecycle.java:285) ~[druid-server-28.0.1-htrunk50.jar:?]
at org.apache.druid.server.QueryResource$QueryResourceQueryResultPusher$1.start(QueryResource.java:512) ~[druid-server-28.0.1-htrunk50.jar:?]
at org.apache.druid.server.QueryResultPusher.push(QueryResultPusher.java:121) ~[druid-server-28.0.1-htrunk50.jar:?]
... 85 more

Start with SegmentMetadataCache.refresh, refreshSegments, refreshSegmentsForDataSource, and the getBundle method shown in the issue; trace how the missing DatasourceBundle exception is handled and requeued. Use the reported broker and indexer logs to reproduce the failure, then verify that a missing bundle no longer causes indefinite metadata-refresh retries.

Written by the indexing model from the issue text.