duckdb / duckdb/duckdb-postgres

Duplicated records when row is updated between transactions

Open
#381 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
372
Forks
105
Avg merge
10h 19m
Merged PRs (30d)
18

Description

### What happens?

We started seeing duplicated rows when extracting data from PostgresDB and noticed that it happens when the row is updated/created when different transaction starts, which seems to be similar to what was reported back in https://github.com/duckdb/duckdb-postgres/issues/163.

I checked our logs with the flag `pg_debug_show_queries=true` after I saw this [change](https://github.com/duckdb/duckdb-postgres/pull/333) and validated all connections are using `BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY` as expected.

However, I noticed `SET TRANSACTION SNAPSHOT 'some-snapshot-id'` is **never called** and neither the expected query before that: `SELECT pg_is_in_recovery(), pg_export_snapshot(), (select count(*) from pg_stat_wal_receiver)`.

This query works when executed directly in our database and provides the following result:

```
SELECT pg_is_in_recovery(), pg_export_snapshot(), (select count(*) from pg_stat_wal_receiver);
pg_is_in_recovery | pg_export_snapshot | count
-------------------+---------------------+-------
f | 0000102C-00400680-1 | 0
```

I suspect the code is not reaching the [part](https://github.com/duckdb/duckdb-postgres/blob/6dabd956065e59b24d615fe03bab84d677fca6df/src/postgres_scanner.cpp#L92-L93) where it gets `pg_export_snapshot` for some reason and therefore affecting the results of the reading operation even when the isolation level is correct. Our workflow has `pg_connection_limit=4`.

### To Reproduce

A bit hard to reproduce since the update operation of a row needs to happen between the start of concurrent transactions (we see this issue happening a couple of times per week), but we're happy to test some code in our workflows.

### OS:

Linux

### PostgreSQL Version:

17.5

### DuckDB Version:

1.4.1

### DuckDB Client:

Python

### Full Name:

Rina Sakurai

### Affiliation:

Mentimeter

### Have you tried this on the latest `main` branch?

- [x] I agree

### Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

- [x] I agree

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.