opensearch-project / opensearch-project/security

LDAP Connection timed out when not using global catalog port in LDAP config

Open
#1,682 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted triaged
Dominant language
Java
Stars
252
Forks
395
Avg merge
1d 11h
Merged PRs (30d)
76

Description

Describe the bug
Hi everybody, I am not sure if this is a bug but I would like to share with you my experience in order to know if anyone else has had the same problem, how and if they have solved it.
I installed opensearch and opensearch-dashboard via helm-chart, rewrote the opensearch configuration to connect my LDAP server to the opensearch instance for authc and authz.
Initially I used port 389, I could easily log in as admin, but as soon as I used my LDAP credentials the dashboard (after at least 30 seconds of loading) reported that I had used an incorrect username and password; the logs were full of connection time out errors:

[2022-03-01T12:58:25,697][WARN ][o.l.r.SearchReferralHandler] [opensearch-test-cluster-master-0] Could not follow referral to ldap://ForestDnsZones.example.com/DC=ForestDnsZones,DC=example,DC=com
org.ldaptive.provider.ConnectionException: javax.naming.CommunicationException: ForestDnsZones.example.com:389 [Root exception is java.net.SocketTimeoutException: Connect timed out]
	at org.ldaptive.provider.jndi.JndiConnectionFactory.createInternal(JndiConnectionFactory.java:90) ~[ldaptive-1.2.3.jar:?]
	at org.ldaptive.provider.jndi.JndiConnectionFactory.createInternal(JndiConnectionFactory.java:21) ~[ldaptive-1.2.3.jar:?]
	at org.ldaptive.provider.AbstractProviderConnectionFactory.create(AbstractProviderConnectionFactory.java:84) ~[ldaptive-1.2.3.jar:?]
	at org.ldaptive.DefaultConnectionFactory$DefaultConnection.open(DefaultConnectionFactory.java:267) ~[ldaptive-1.2.3.jar:?]
	at org.ldaptive.referral.AbstractReferralHandler.followReferral(AbstractReferralHandler.java:156) [ldaptive-1.2.3.jar:?]
	at org.ldaptive.referral.AbstractReferralHandler.handle(AbstractReferralHandler.java:221) [ldaptive-1.2.3.jar:?]
	at org.ldaptive.referral.SearchReferralHandler$SearchReferenceHandler.handle(SearchReferralHandler.java:268) [ldaptive-1.2.3.jar:?]
	at org.ldaptive.referral.SearchReferralHandler$SearchReferenceHandler.handle(SearchReferralHandler.java:155) [ldaptive-1.2.3.jar:?]
	at org.ldaptive.AbstractOperation.executeHandlers(AbstractOperation.java:186) [ldaptive-1.2.3.jar:?]
	at org.ldaptive.SearchOperation.readResult(SearchOperation.java:152) [ldaptive-1.2.3.jar:?]
	at org.ldaptive.SearchOperation.executeSearch(SearchOperation.java:104) [ldaptive-1.2.3.jar:?]
	at org.ldaptive.SearchOperation.invoke(SearchOperation.java:85) [ldaptive-1.2.3.jar:?]
	at org.ldaptive.SearchOperation.invoke(SearchOperation.java:15) [ldaptive-1.2.3.jar:?]
	at org.ldaptive.AbstractOperation.execute(AbstractOperation.java:126) [ldaptive-1.2.3.jar:?]
	at com.amazon.dlic.auth.ldap.util.LdapHelper$1.run(LdapHelper.java:67) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at com.amazon.dlic.auth.ldap.util.LdapHelper$1.run(LdapHelper.java:56) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at java.security.AccessController.doPrivileged(AccessController.java:554) [?:?]
	at com.amazon.dlic.auth.ldap.util.LdapHelper.search(LdapHelper.java:56) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.resolveNestedRoles(LDAPAuthorizationBackend.java:1021) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.fillRoles(LDAPAuthorizationBackend.java:894) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at org.opensearch.security.auth.BackendRegistry.authz(BackendRegistry.java:635) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at org.opensearch.security.auth.BackendRegistry.access$000(BackendRegistry.java:82) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at org.opensearch.security.auth.BackendRegistry$9.call(BackendRegistry.java:675) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at org.opensearch.security.auth.BackendRegistry$9.call(BackendRegistry.java:668) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4875) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3951) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4870) [guava-25.1-jre.jar:?]
	at org.opensearch.security.auth.BackendRegistry.authcz(BackendRegistry.java:668) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:471) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:188) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at org.opensearch.security.filter.SecurityRestFilter.access$000(SecurityRestFilter.java:72) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:123) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:306) [opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:392) [opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:235) [opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-1.2.4.0.jar:1.2.4.0]
	at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:361) [opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:440) [opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:351) [opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-1.2.4.jar:1.2.4]
	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-1.2.4.jar:1.2.4]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-1.2.4.jar:1.2.4]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.72.Final.jar:4.1.72.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.naming.CommunicationException: ForestDnsZones.eample.com:389
	at com.sun.jndi.ldap.Connection.<init>(Connection.java:244) ~[?:?]
	at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137) ~[?:?]
	at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1616) ~[?:?]
	at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2847) ~[?:?]
	at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:348) ~[?:?]
	at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxFromUrl(LdapCtxFactory.java:225) ~[?:?]
	at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:189) ~[?:?]
	at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:243) ~[?:?]
	at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:154) ~[?:?]
	at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:84) ~[?:?]
	at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:719) ~[?:?]
	at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:305) ~[?:?]
	at javax.naming.InitialContext.init(InitialContext.java:236) ~[?:?]
	at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) ~[?:?]
	at org.ldaptive.provider.jndi.JndiConnectionFactory.createInternal(JndiConnectionFactory.java:87) ~[ldaptive-1.2.3.jar:?]
	... 98 more
Caused by: java.net.SocketTimeoutException: Connect timed out
	at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
	at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]
	at java.net.Socket.connect(Socket.java:648) ~[?:?]
	at com.sun.jndi.ldap.Connection.createSocket(Connection.java:327) ~[?:?]
	at com.sun.jndi.ldap.Connection.<init>(Connection.java:223) ~[?:?]
	at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137) ~[?:?]
	at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1616) ~[?:?]
	at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2847) ~[?:?]
	at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:348) ~[?:?]
	at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxFromUrl(LdapCtxFactory.java:225) ~[?:?]
	at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:189) ~[?:?]
	at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:243) ~[?:?]
	at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:154) ~[?:?]
	at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:84) ~[?:?]
	at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:719) ~[?:?]
	at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:305) ~[?:?]
	at javax.naming.InitialContext.init(InitialContext.java:236) ~[?:?]
	at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) ~[?:?]
	at org.ldaptive.provider.jndi.JndiConnectionFactory.createInternal(JndiConnectionFactory.java:87) ~[ldaptive-1.2.3.jar:?]

Finally I solved the problem by changing the LDAP-Server port to the global-catalog port, 3268 and 3269 (fort TLS).
I already reported the problem and the findig to guys of the helm-chart repo, but I just thought that was interessant to share my experience with you too.
Please feel free to ask any context, details and question.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the LDAPAuthorizationBackend and LdapHelper stack-trace entries, then reproduce the Helm-based OpenSearch LDAP setup using port 389. Compare the configured LDAP endpoint with the referral to ForestDnsZones.example.com and define the expected referral behavior and a regression test before changing code.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
authentication, authorization, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.