Unresolved memcached host results in druid not starting up
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Druid fails to start when a memcached host can not be resolved.
### Affected Version
v32.0.0
### Description
I am setting up a 2-layer cache on my test druid system.
When setting it up, i had a typo in my memcached hosts config, so the address was not resolvable.
The result was that druid wouldn't start up.
This exact behavior would also happen, when a memcached pod would get re-scheduled at the same time that a druid pod is re-scheduled.
That makes it undesirable to have druid not starting up when remote cache is not available.
Under normal circumstances the memcached would be available very quickly again, however there can be issues like volumes not attachable when the cache would not be available quickly again.
Please include as much detailed information about the problem as possible.
- Cluster size: Doesn't matter
- Configurations in use
- Steps to reproduce the problem:
- configure hybrid mode cache with memcached as L2
- set dns to something that is not resolvable
- see error in logs
- The error message or stack traces encountered. Providing more context, such as nearby log messages or even entire logs, can be helpful.
- Any debugging that you have already done
logs:
```
Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:
1) Error in custom provider, java.nio.channels.UnresolvedAddressException
while locating org.apache.druid.client.cache.CacheProvider annotated with interface org.apache.druid.guice.annotations.Global
at org.apache.druid.guice.CacheModule.configure(CacheModule.java:51) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliHistorical$$Lambda$211/0x0000001001346d38 -> org.apache.druid.guice.CacheModule)
while locating org.apache.druid.client.cache.Cache
for the 1st parameter of org.apache.druid.client.cache.CacheMonitor.setCache(CacheMonitor.java:48)
while locating org.apache.druid.client.cache.CacheMonitor
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:112) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:112) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler
at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:97) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)
1 error
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:88)
at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:69)
at org.apache.druid.cli.Main.main(Main.java:112)
Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:
1) Error in custom provider, java.nio.channels.UnresolvedAddressException
while locating org.apache.druid.client.cache.CacheProvider annotated with interface org.apache.druid.guice.annotations.Global
at org.apache.druid.guice.CacheModule.configure(CacheModule.java:51) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliHistorical$$Lambda$211/0x0000001001346d38 -> org.apache.druid.guice.CacheModule)
while locating org.apache.druid.client.cache.Cache
for the 1st parameter of org.apache.druid.client.cache.CacheMonitor.setCache(CacheMonitor.java:48)
while locating org.apache.druid.client.cache.CacheMonitor
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:112) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:112) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler
at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:97) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)
1 error
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
at com.google.inject.Guice.createInjector(Guice.java:99)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at org.apache.druid.initialization.ExtensionInjectorBuilder.build(ExtensionInjectorBuilder.java:49)
at org.apache.druid.initialization.ServerInjectorBuilder.build(ServerInjectorBuilder.java:118)
at org.apache.druid.initialization.ServerInjectorBuilder.makeServerInjector(ServerInjectorBuilder.java:73)
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:85)
... 2 more
Caused by: java.nio.channels.UnresolvedAddressException
at java.base/sun.nio.ch.Net.checkAddress(Net.java:149)
at java.base/sun.nio.ch.Net.checkAddress(Net.java:157)
at java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:816)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:839)
at net.spy.memcached.MemcachedConnection.createConnections(MemcachedConnection.java:479)
at net.spy.memcached.MemcachedConnection.(MemcachedConnection.java:369)
at net.spy.memcached.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:253)
at net.spy.memcached.MemcachedClient.setupConnection(MemcachedClient.java:350)
at net.spy.memcached.MemcachedClient.(MemcachedClient.java:297)
at net.spy.memcached.MemcachedClient.(MemcachedClient.java:229)
at org.apache.druid.client.cache.MemcachedCache$5.get(MemcachedCache.java:365)
at org.apache.druid.client.cache.MemcachedCache$5.get(MemcachedCache.java:360)
at org.apache.druid.client.cache.MemcacheClientPool.(MemcacheClientPool.java:68)
at org.apache.druid.client.cache.MemcachedCache.create(MemcachedCache.java:358)
at org.apache.druid.client.cache.MemcachedCacheProvider.get(MemcachedCacheProvider.java:27)
at org.apache.druid.client.cache.MemcachedCacheProvider.get(MemcachedCacheProvider.java:22)
at org.apache.druid.client.cache.HybridCacheProvider.get(HybridCacheProvider.java:51)
at org.apache.druid.client.cache.HybridCacheProvider.get(HybridCacheProvider.java:27)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72)
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at org.apache.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:82)
at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:120)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054)
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:118)
at org.apache.druid.server.metrics.MetricsModule$$FastClassByGuice$$99ddce1b.invoke()
at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at org.apache.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
... 10 more
```
Contributor guide
Research direction
Start with org.apache.druid.client.cache.MemcachedCache.create and MemcachedCacheProvider.get, then trace the cache binding from CacheModule.java:51. Reproduce the unresolved-host configuration from the issue and verify that Druid can start when memcached cannot be resolved, while preserving the existing cache setup behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100