logdna / logdna/ruby

Logs failing to flush in `Logdna::Client` due to concurrency issue in rails

Aperta
#46 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Lingua principale
Ruby
Stelle
19
Fork
18
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Found that any uses of Logdna::Client within the boot sequence of rails results in subsequent logs never flushing.

For background, using logdna 1.5.0 w/ rails 6.1.7.5 on ruby 3.2.2. Configuring Logdna::Ruby following the instructions in TAMING RUBY ON RAILS LOGGING WITH LOGRAGE AND MEZMO in config/application.rb. As long as Rails.logger isn't used within the subsequent application setup steps, then logging works as expected. But if any other subsequent step of the application setup process attempts to log, then all subsequent logs fail to be emitted.

Tracked this down to Logdna::Client#schedule_flush. By adding logging, I found that the associated Concurrent::ScheduledTask in @scheduled_flush remains in the pending state indefinitely. Further logging shows that the ScheduledTask is associated with a Concurrent::CachedThreadPool that contains a single thread and that thread has is marked as dead.

Guessing that as part of the rails boot up process, the default executor is terminated and therefore the @scheduled_flush never completes. Not that using the @work_thread_pool thread pool doesn't fix the issue; the same behavior is observed. By not logging anything during rails configurations/boot, a ScheduledTask isn't created while the application is still booting, and therefore this issue doesn't occur.

I think this issue could be addressed by tracking the time at which @scheduled_flush is created and then in subsequent invocations of schedule_flush checking if it has been pending far longer than expected. If it is sufficiently stale, then the outstanding ScheduledTask would be cancelled and a new one created. There may be more robust/elegant ways to address this because I'm not particularly familiar with ruby/rails thread management.

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia da Logdna::Client#schedule_flush e analizza come @scheduled_flush e @work_thread_pool utilizzano le attività pianificate di Concurrent durante l'avvio di Rails. Riproduci il problema con la configurazione descritta di Rails 6.1.7.5 e Ruby 3.2.2, includendo una chiamata di log da config/application.rb; il lavoro è completato quando i log continuano a essere scaricati dopo l'avvio, anche quando vengono prodotti log di configurazione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rails, ruby
Ambito
backend, observability-sre
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.