new s3 sink output format with less data for active-active multi cluster consistency checker
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
Is your feature request related to a problem?
multi-cluster-consistency-checker will detect data loss, data redundant, data inconsistency and lww violation, which only needs the record's PK, commit_ts, origin_ts and the consistency of value of other columns.
In order to reduce the data transfer, here request a new s3 sink output format which only contains PK, commit_ts, origin_ts and checksum of other columns.
Describe the feature you'd like
The file is in JSON format and contains multiple lines.
{"pk_data": {"pk_col_1": "0", "pk_col_2": 0, ...}, commit_ts: 0, origin_ts: 0, checksum: "0"}\r\n
Describe alternatives you've considered
No response
Teachability, Documentation, Adoption, Migration Strategy
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the S3 sink and the multi-cluster-consistency-checker entry points in TiCDC. Review how the current S3 output is serialized, then define the JSON-lines output containing PK data, commit_ts, origin_ts, and a checksum for other columns. Done means the new format reduces transferred data while preserving the fields needed by the consistency checker.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100