temporalio / temporalio/temporal

Support for loading history events using size of batches rather than count

Open
#677 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

difficulty: hard enhancement operations
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.
Temporal writes event to historystore in a batch which can be of variable sizes. When we query
for a page of event we set the page size based on number of batches we read from database.
This could be a problem if user is writing very large payloads and single batch is large in size.
We may end of loading unnecessary events which we have to throw away when returning the
events back to client based on the page size set by the caller.

Describe the solution you'd like
We need to design a solution which allows us to load events based on size rather than count.

Describe alternatives you've considered
Another alternate is to still load on count but have throttling in place using size of payloads
loaded from DB. Issue #675

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the history-store query that sets page size from the number of batches read. Compare loading by batch count with the proposed payload-size approach, and review issue #675 for the throttling alternative. Done means the design prevents unnecessarily loading large event batches while preserving the caller's page-size behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.