apache / apache/druid

Unresolved memcached host results in druid not starting up

Open
#17,789 2 comments 1 reaction 0 assignees View on GitHub
Area - Cache
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Druid fails to start when a memcached host can not be resolved.

### Affected Version

v32.0.0

### Description
I am setting up a 2-layer cache on my test druid system.
When setting it up, i had a typo in my memcached hosts config, so the address was not resolvable.
The result was that druid wouldn't start up.
This exact behavior would also happen, when a memcached pod would get re-scheduled at the same time that a druid pod is re-scheduled.
That makes it undesirable to have druid not starting up when remote cache is not available.
Under normal circumstances the memcached would be available very quickly again, however there can be issues like volumes not attachable when the cache would not be available quickly again.

Please include as much detailed information about the problem as possible.
- Cluster size: Doesn't matter
- Configurations in use
- Steps to reproduce the problem:
- configure hybrid mode cache with memcached as L2
- set dns to something that is not resolvable
- see error in logs
- The error message or stack traces encountered. Providing more context, such as nearby log messages or even entire logs, can be helpful.
- Any debugging that you have already done

logs:
```
Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error in custom provider, java.nio.channels.UnresolvedAddressException
while locating org.apache.druid.client.cache.CacheProvider annotated with interface org.apache.druid.guice.annotations.Global
at org.apache.druid.guice.CacheModule.configure(CacheModule.java:51) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliHistorical$$Lambda$211/0x0000001001346d38 -> org.apache.druid.guice.CacheModule)
while locating org.apache.druid.client.cache.Cache
for the 1st parameter of org.apache.druid.client.cache.CacheMonitor.setCache(CacheMonitor.java:48)
while locating org.apache.druid.client.cache.CacheMonitor
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:112) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:112) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler
at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:97) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

1 error
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:88)
at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:69)
at org.apache.druid.cli.Main.main(Main.java:112)
Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error in custom provider, java.nio.channels.UnresolvedAddressException
while locating org.apache.druid.client.cache.CacheProvider annotated with interface org.apache.druid.guice.annotations.Global
at org.apache.druid.guice.CacheModule.configure(CacheModule.java:51) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliHistorical$$Lambda$211/0x0000001001346d38 -> org.apache.druid.guice.CacheModule)
while locating org.apache.druid.client.cache.Cache
for the 1st parameter of org.apache.druid.client.cache.CacheMonitor.setCache(CacheMonitor.java:48)
while locating org.apache.druid.client.cache.CacheMonitor
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:112) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:112) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler
at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:97) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

1 error
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
at com.google.inject.Guice.createInjector(Guice.java:99)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at org.apache.druid.initialization.ExtensionInjectorBuilder.build(ExtensionInjectorBuilder.java:49)
at org.apache.druid.initialization.ServerInjectorBuilder.build(ServerInjectorBuilder.java:118)
at org.apache.druid.initialization.ServerInjectorBuilder.makeServerInjector(ServerInjectorBuilder.java:73)
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:85)
... 2 more
Caused by: java.nio.channels.UnresolvedAddressException
at java.base/sun.nio.ch.Net.checkAddress(Net.java:149)
at java.base/sun.nio.ch.Net.checkAddress(Net.java:157)
at java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:816)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:839)
at net.spy.memcached.MemcachedConnection.createConnections(MemcachedConnection.java:479)
at net.spy.memcached.MemcachedConnection.(MemcachedConnection.java:369)
at net.spy.memcached.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:253)
at net.spy.memcached.MemcachedClient.setupConnection(MemcachedClient.java:350)
at net.spy.memcached.MemcachedClient.(MemcachedClient.java:297)
at net.spy.memcached.MemcachedClient.(MemcachedClient.java:229)
at org.apache.druid.client.cache.MemcachedCache$5.get(MemcachedCache.java:365)
at org.apache.druid.client.cache.MemcachedCache$5.get(MemcachedCache.java:360)
at org.apache.druid.client.cache.MemcacheClientPool.(MemcacheClientPool.java:68)
at org.apache.druid.client.cache.MemcachedCache.create(MemcachedCache.java:358)
at org.apache.druid.client.cache.MemcachedCacheProvider.get(MemcachedCacheProvider.java:27)
at org.apache.druid.client.cache.MemcachedCacheProvider.get(MemcachedCacheProvider.java:22)
at org.apache.druid.client.cache.HybridCacheProvider.get(HybridCacheProvider.java:51)
at org.apache.druid.client.cache.HybridCacheProvider.get(HybridCacheProvider.java:27)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72)
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at org.apache.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:82)
at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:120)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054)
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:118)
at org.apache.druid.server.metrics.MetricsModule$$FastClassByGuice$$99ddce1b.invoke()
at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at org.apache.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
... 10 more
```

Contributor guide

Open the contributing guide

Research direction

Start with org.apache.druid.client.cache.MemcachedCache.create and MemcachedCacheProvider.get, then trace the cache binding from CacheModule.java:51. Reproduce the unresolved-host configuration from the issue and verify that Druid can start when memcached cannot be resolved, while preserving the existing cache setup behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.