elastic / elastic/elasticsearch-ruby

Closing open connections to Elasticsearch

Abierto
#2,389 0 comentarios 0 reacciones 0 asignados Ver en GitHub
question
Lenguaje dominante
Ruby
Estrellas
2k
Forks
615
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

We are using `elasticsearch-ruby` through the `chewy` gem, and we import documents in a Sidekiq worker. This means each Sidekiq thread that accesses Elasticsearch has its own instance of the `Elasticsearch::Client`, and when a (possibly unrelated) Sidekiq job fails, that worker thread is closed and a new thread is opened in the same process.

One thing we noticed though, is that the underlying Elasticsearch connections are only closed when Ruby's garbage collector collects the dead thread's `Elasticsearch::Client` instance, which seems to be the cause of a file descriptor leak in our application.

We think we have found a way to close these connections by adding the following code to the error handler in a custom Sidekiq middleware:
```ruby
Chewy.client.transport.transport.connections.each do |connection|
# This bit of code is tailored for the HTTPClient Faraday adapter
connection.connection.app.instance_variable_get(:@client)&.reset_all
end
```

However, this piece of code breaks multiple layers of abstractions, going through `chewy`, `elasticsearch`, `elasticsearch-transport`, `faraday` and `faraday-httpclient`, even accessing an otherwise unexposed instance variable at one point.

Is there a better way of closing connections to Elasticsearch? Are we missing something obvious about their lifecycle?

---

Digging into it, my understanding of the issue is that neither `elasticsearch` nor `elasticsearch-transport` provide a method to close connections.

It looks like `faraday` has `Faraday::Connection#close` but that appears to not actually be implemented in most adapters, and in particular not in the `faraday-httpclient` adapter that ends up being used in our app.

Of course, I may have missed something, and would be glad to know what if that's the case!

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.