lbryio / lbryio/lbry-sdk

Connection to ElasticSearch running inside Docker fails (MacOS)

Open
#3,708 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.2k
Forks
493
PR merge metrics
No merged PRs in 30d

Description

I am running lbry-sdk integration tests locally on MacOS, with elastic-search inside Docker. I used the lbry-sdk/Makefile command to launch it:

```
elastic-docker:
docker run -d -v lbryhub:/usr/share/elasticsearch/data -p 9200:9200 -p 9300:9300 -e"ES_JAVA_OPTS=-Xms512m -Xmx512m" -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.12.1
```

In order to connect herald -> elastic-search, I had to disable the SetSniff() option.

```
diff --git a/server/server.go b/server/server.go
index 415b728..7706e72 100644
--- a/server/server.go
+++ b/server/server.go
@@ -210,7 +210,7 @@ func MakeHubServer(ctx context.Context, args *Args) *Server {
if !args.DisableEs {
esUrl := args.EsHost + ":" + args.EsPort
opts := []elastic.ClientOptionFunc{
- elastic.SetSniff(true),
+ elastic.SetSniff(false),
elastic.SetSnifferTimeoutStartup(time.Second * 60),
elastic.SetSnifferTimeout(time.Second * 60),
elastic.SetURL(esUrl),
```

Probably the clearest explanation:
https://github.com/olivere/elastic/issues/1138

More similar issues:
https://github.com/olivere/elastic/issues?q=is%3Aissue+docker+SetSniff+SetHealthCheck

***Making a --nosniff option MAY NOT be the best solution here.*** I found the following comment describes how to fix this using the ```network.publish_host``` option of ElasticSearch. https://github.com/olivere/elastic/issues/824#issuecomment-474416267

Example:
docker run --env network.publish_host=127.0.0.1 ...

***I tested "--env network.publish_host=127.0.0.1" and it worked for me.***

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#advanced-network-settings

Symbolic addresses like ```network.publish_host=_local_``` are supported.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#network-interface-values

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with server/server.go and the elastic-docker target in Makefile, then trace how the Elastic client uses SetSniff when connecting to the Dockerized Elasticsearch instance. Reproduce the MacOS integration-test setup and compare the reported sniffer and network.publish_host behaviors. Done means the connection works through the documented setup without a local source change or an unexplained workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, elasticsearch, go
Domain
backend, infrastructure, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.