coinbase / coinbase/temporal-ruby

Timer timing issue

Aperta
#201 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Ruby
Stelle
287
Fork
113
Merge medio
6g 11h
PR unite (30g)
1

Descrizione

# temporal-ruby Timer timing issue

## Scenario:

- Timer started (30 minutes)
- Sleep started by workflow code (15 minute)
- Worker goes down
- Sleep ends
- Timer fires.
- Worker comes back up
- Worker receives a history containing the sleep finishing followed by timer firing.

The workflow thus receives two TIMER_FIRED events in its history at once. Now what should happen?

## What’s happening in temporal-ruby today

It processes the timers in order. Sleep first, then the other timer. The workflow completes first, and it’s not allowed to do anything after the workflow has completed. Thus, if the timer issues a command, the task will fail with `Temporal::WorkflowAlreadyCompletingError`.

### Attempted solution

My first attempt was to early-out in `state_manager.rb`

history_window.events.each do |event|
apply_event(event)
break if if workflow_finished?
end

This fixed my problem and is logical; the workflow will execute the same as it would have if the worker had never gone down. But, it will silently drop events.
I asked Maxim what other SDKs do…

## Other SDKs’ behaviors

According to Maxim:

> Temporal processes new events in batches. One batch per workflow task. It applies all the events before running workflow code. So in case of Java or Go, for example the Futures that correspond to timer will be set to ready before running any workflow threads. This way workflow can check if any of these Futures are ready to decide what to do next.
>
> Signals have similar properties, but they use callbacks in Java and some other SDKs. Temporal SDK schedules callback threads before running workflow threads. This way workflow cannot complete before the callback threads were called.

> Then in case of Java SDK it is going to make the main workflow thread eligible to run. But it still will run only after all other threads executed and all other Futures resolved.
> Without this functionality you guaranteed to lose signals if they are delivered as callbacks.
>

Rust core does the same

[Java implementation](https://github.com/temporalio/sdk-java/blob/e620a9fcca29a41a2b9ce3aefb99be5fb6ce2630/temporal-sdk/src/main/java/io/temporal/internal/sync/DeterministicRunnerImpl.java#L500) - we can a see that callbacks (timer in this case) have a higher priority than the main thread (which is awoken by the sleep in this case).

## What now?

Changing temporal-ruby's behavior is obviously somewhat tricky and would cause version changes if not done in a version-safe way. We might take this fix on but want to work with the community.
* Should we fix this?
* Implementation suggestions for the fix itself?
* Version safety. I'm guessing we should add a configuration that changes this behavior that folks can opt into.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da state_manager.rb e dal ciclo di elaborazione degli eventi history_window descritto nell’issue. Riproduci lo scenario del workflow in cui il completamento di sleep e l’attivazione del timer arrivano nello stesso batch della cronologia, quindi confronta il comportamento attuale con quello citato degli SDK Java e Rust. L’issue non ha ancora un’implementazione concordata né una definizione di completamento.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
ruby
Ambito
backend, distributed-systems
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.