Subscriptions might lose data
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 8.7k
- Forks
- 374
- Avg merge
- 11d 16h
- Merged PRs (30d)
- 1
Description
I'm currently implementing CDC in a web application on top of postgres.
I want to:
- Fetch the users table
- Be notified whenever the users table changes and update my local copy
But obviously, I also need to be careful of the race condition: What happens if the data is modified between my fetch and subscribe calls?
The "right answer", as far as I can tell, is to get the WAL log location (lsn) along with my fetch query, and then subscribe to all changes from that location onwards. For example, to fetch:
let startLsn
{
await sql.begin('ISOLATION LEVEL REPEATABLE READ', async sql => {
const lsnResult = await sql`SELECT pg_current_wal_lsn() as lsn`;
startLsn = lsnResult[0].lsn
const result = await sql<User[]>`SELECT * FROM users`;
result.forEach((user) => {
users.set(user.id, user);
});
})
}
But then to subscribe I need to pass the LSN in. I want to make this query:
sql`START_REPLICATION SLOT users_cache_slot LOGICAL ${startLsn} (proto_version '1', publication_names 'users_pub')`
Unfortunately, the subscription code in postgres can't do this as far as I can tell?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by locating the subscription implementation and its PostgreSQL logical-replication path, then determine whether it can accept the supplied LSN for START_REPLICATION; done means replication can begin from that LSN without losing changes between the fetch and subscription.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, typescript
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100