temporalio / temporalio/temporal

Compress history_node->data cell

Open
#10,307 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Some data cells can be 2M or so large, which isn't great, considering this is just proto3 data, written once, rarely read. They should be compressed by default, perhaps with some minimum size that is worth the effort.
Example from a production workload:

Image

It seems I can use this:

The PayloadCodec interface:
type PayloadCodec interface {
    Encode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error)
    Decode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error)
}

If approved, I'll give it a try.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the PayloadCodec interface and the history_node data-cell write and read paths. Determine where a minimum compression threshold belongs and verify that default compression preserves decoding behavior; done means large cells are compressed while smaller cells avoid unnecessary work.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.