elastic / elastic/elasticsearch-ruby

Closing open connections to Elasticsearch

Offen
#2,389 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
question
Vorherrschende Sprache
Ruby
Sterne
2k
Forks
615
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

We are using `elasticsearch-ruby` through the `chewy` gem, and we import documents in a Sidekiq worker. This means each Sidekiq thread that accesses Elasticsearch has its own instance of the `Elasticsearch::Client`, and when a (possibly unrelated) Sidekiq job fails, that worker thread is closed and a new thread is opened in the same process.

One thing we noticed though, is that the underlying Elasticsearch connections are only closed when Ruby's garbage collector collects the dead thread's `Elasticsearch::Client` instance, which seems to be the cause of a file descriptor leak in our application.

We think we have found a way to close these connections by adding the following code to the error handler in a custom Sidekiq middleware:
```ruby
Chewy.client.transport.transport.connections.each do |connection|
# This bit of code is tailored for the HTTPClient Faraday adapter
connection.connection.app.instance_variable_get(:@client)&.reset_all
end
```

However, this piece of code breaks multiple layers of abstractions, going through `chewy`, `elasticsearch`, `elasticsearch-transport`, `faraday` and `faraday-httpclient`, even accessing an otherwise unexposed instance variable at one point.

Is there a better way of closing connections to Elasticsearch? Are we missing something obvious about their lifecycle?

---

Digging into it, my understanding of the issue is that neither `elasticsearch` nor `elasticsearch-transport` provide a method to close connections.

It looks like `faraday` has `Faraday::Connection#close` but that appears to not actually be implemented in most adapters, and in particular not in the `faraday-httpclient` adapter that ends up being used in our app.

Of course, I may have missed something, and would be glad to know what if that's the case!

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.