indeedeng / indeedeng/iwf

[perf] fix double serialization for continueAsNew input (IWF-140)

Open
#414 1 comment 0 reactions 0 assignees View on GitHub
2nd important in-review medium
Dominant language
Go
Stars
661
Forks
63
PR merge metrics
No merged PRs in 30d

Description

Right now the input is stored like this in the history:
```
[{"checksum":"\ufffd^\ufffd`\ufffd\ufffd\ufffd\ufffd\u0014_H\u0016\ufffd\ufffd\ufffd=","jsonData":"{\"StatesToStartFromBeginning\":[{\"stateId\":\"PersistRuleState\",\"stateInput\":{\"data\":\"{\\\"createdByUserId\\\":\\\"mwhitty\\\",\\\"shouldApplyToExistingJobs\\\":true,\\\"tagsToAdd\\\":[\\\"tag1\\\",\\\"tag2\\\"],\\\"statusToChange\\\":\\\"OPEN\\\",\\\"campaignToAdd\\\":\\\"CAMPAIGN-X\\\",\\\"ownerToChange\\\":\\\"qlong\\\",\\\"userIdsToNotify\\\":[\\\"user1\\\",\\\"user2\\\"],\\\"userIdsToGrantEditAccess\\\":[\\\"user3\\\",\\\"user4\\\"]}\",\"encoding\":\"springJackson\"}}],\"StateExecutionsToResume\":{},\"InterStateChannelReceived\":{},\"SignalsReceived\":{},\"StateExecutionCounterInfo\":{\"StateIdStartedCount\":{\"PersistRuleState\":50},\"StateIdCurrentlyExecutingCount\":{},\"TotalCurrentlyExecutingCount\":0},\"StateOutputs\":[],\"StaleSkipTimerSignals\":null,\"DataObjects\":[{\"key\":\"TotalPages\",\"value\":{\"data\":\"0\",\"encoding\":\"springJackson\"}}],\"SearchAttributes\":null}","totalPages":1}]
```
You can see a lot of `\\` in it which is very wasteful and hard to read.

This is because this `DumpWorkflowInternal` activity has serialized the data that is already JSON serialized.
https://github.com/indeedeng/iwf/blob/ef706abc8217d0d625d56d5b315c12c23dd4ba55/service/interpreter/activityImpl.go#L264
We need to improve this.

---------------------

Spent two days into this, and realized that JSON is not going to work for saving the double serialization. This is because the data that we are storing is not valid JSON because of pagination, hence we cannot use json.RawMessage.

The best way is to use proto based serialization like gRPC.

Defer this for now until we know it's necessary to do this

Contributor guide

Open the contributing guide

Research direction

Start with service/interpreter/activityImpl.go at the DumpWorkflowInternal code around line 264 and inspect how its already-serialized data is handled. The issue notes that JSON is incompatible with pagination and defers the work toward proto-based serialization; done would require an agreed serialization approach that avoids double serialization while preserving the stored data.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.