getting xid of each insert, update and delete statements
- Dominant language
- Go
- Stars
- 546
- Forks
- 83
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 1
Description
I want to create a change data capture service which listens to postgresql and stream changes to kafka. As I read in this [article](https://www.postgresql.eu/events/pgconfeu2023/sessions/session/4773/slides/427/A%20journey%20into%20postgresql%20logical%20replication.pdf), lsn of different transaction commands, during concurrent writes, are not necessarily in order (but commit messages are ordered). So I wanted to group messages by xid and write them to kafka in batches.
However, when I inspect xid of insert, update and delete statements, they are always zero. On solution that came to my mind was to set `inStream=true` in BeginMessage phase and set it to `false` in CommitMessage phase. I don't know if it is a valid action and I don't see capturing this events in this project example. This action is done during StreamStartMessageV2 and StreamStopMessageV2 which my code (and also the example code) does not produce it.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files or tests. Begin with the example code's BeginMessage, CommitMessage, StreamStartMessageV2, and StreamStopMessageV2 paths; verify how xid is exposed for row changes. Done means documenting or reproducing a supported way to group INSERT, UPDATE, and DELETE messages by xid and confirming the Kafka batching expectation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka, postgresql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100