[perf] fix double serialization for continueAsNew input (IWF-140)
- Dominant language
- Go
- Stars
- 661
- Forks
- 63
- PR merge metrics
- No merged PRs in 30d
Description
Right now the input is stored like this in the history:
```
[{"checksum":"\ufffd^\ufffd`\ufffd\ufffd\ufffd\ufffd\u0014_H\u0016\ufffd\ufffd\ufffd=","jsonData":"{\"StatesToStartFromBeginning\":[{\"stateId\":\"PersistRuleState\",\"stateInput\":{\"data\":\"{\\\"createdByUserId\\\":\\\"mwhitty\\\",\\\"shouldApplyToExistingJobs\\\":true,\\\"tagsToAdd\\\":[\\\"tag1\\\",\\\"tag2\\\"],\\\"statusToChange\\\":\\\"OPEN\\\",\\\"campaignToAdd\\\":\\\"CAMPAIGN-X\\\",\\\"ownerToChange\\\":\\\"qlong\\\",\\\"userIdsToNotify\\\":[\\\"user1\\\",\\\"user2\\\"],\\\"userIdsToGrantEditAccess\\\":[\\\"user3\\\",\\\"user4\\\"]}\",\"encoding\":\"springJackson\"}}],\"StateExecutionsToResume\":{},\"InterStateChannelReceived\":{},\"SignalsReceived\":{},\"StateExecutionCounterInfo\":{\"StateIdStartedCount\":{\"PersistRuleState\":50},\"StateIdCurrentlyExecutingCount\":{},\"TotalCurrentlyExecutingCount\":0},\"StateOutputs\":[],\"StaleSkipTimerSignals\":null,\"DataObjects\":[{\"key\":\"TotalPages\",\"value\":{\"data\":\"0\",\"encoding\":\"springJackson\"}}],\"SearchAttributes\":null}","totalPages":1}]
```
You can see a lot of `\\` in it which is very wasteful and hard to read.
This is because this `DumpWorkflowInternal` activity has serialized the data that is already JSON serialized.
https://github.com/indeedeng/iwf/blob/ef706abc8217d0d625d56d5b315c12c23dd4ba55/service/interpreter/activityImpl.go#L264
We need to improve this.
---------------------
Spent two days into this, and realized that JSON is not going to work for saving the double serialization. This is because the data that we are storing is not valid JSON because of pagination, hence we cannot use json.RawMessage.
The best way is to use proto based serialization like gRPC.
Defer this for now until we know it's necessary to do this
Contributor guide
Research direction
Start with service/interpreter/activityImpl.go at the DumpWorkflowInternal code around line 264 and inspect how its already-serialized data is handled. The issue notes that JSON is incompatible with pagination and defers the work toward proto-based serialization; done would require an agreed serialization approach that avoids double serialization while preserving the stored data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100