temporalio / temporalio/temporal

Improve record activity started implementation

Open
#8,101 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.
RecordActivityStarted implementation today loads the ActivityScheduledEvent while holding the workflow lock. Also the event is not added to the events cache when the activity is being scheduled. This increases the time workflow is locked unnecessarily (because history is immutable and can be accessed without workflow being locked).

When workflow schedules large number of activities at the same time, the transfer task for dispatching the activity (and RecordActivityStarted will be invoked on sync match) will compete with each other for the workflow lock and this additional locking makes the contention worse and task will start to fail, backoff and retry.

Of course we can improve the task processing part to reduce/avoid lock contention, but fixing the RecordActivityStarted implementation is very straightforward and a low hanging fruit.

Describe the solution you'd like

  • Read ActivityScheduledEvent outside workflow lock
  • Cache ActivityScheduledEvent

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the RecordActivityStarted implementation and the code that loads ActivityScheduledEvent. Trace how the workflow lock and events cache are used during activity scheduling, then verify that history access occurs outside the lock and the event is cached without changing behavior. No specific files or tests are named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.