jackc / jackc/pglogrepl

getting xid of each insert, update and delete statements

Open
#66 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
546
Forks
83
Avg merge
2d 21h
Merged PRs (30d)
1

Description

I want to create a change data capture service which listens to postgresql and stream changes to kafka. As I read in this [article](https://www.postgresql.eu/events/pgconfeu2023/sessions/session/4773/slides/427/A%20journey%20into%20postgresql%20logical%20replication.pdf), lsn of different transaction commands, during concurrent writes, are not necessarily in order (but commit messages are ordered). So I wanted to group messages by xid and write them to kafka in batches.
However, when I inspect xid of insert, update and delete statements, they are always zero. On solution that came to my mind was to set `inStream=true` in BeginMessage phase and set it to `false` in CommitMessage phase. I don't know if it is a valid action and I don't see capturing this events in this project example. This action is done during StreamStartMessageV2 and StreamStopMessageV2 which my code (and also the example code) does not produce it.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files or tests. Begin with the example code's BeginMessage, CommitMessage, StreamStartMessageV2, and StreamStopMessageV2 paths; verify how xid is exposed for row changes. Done means documenting or reproducing a supported way to group INSERT, UPDATE, and DELETE messages by xid and confirming the Kafka batching expectation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka, postgresql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.