jackc / jackc/pgx

Using pgx with HAProxy and dead node returns query that is missing data but has no errors

Open
#1,130 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14.3k
Forks
1.1k
Avg merge
6d 9h
Merged PRs (30d)
11

Description

I'm trying to use `pgx` with `HAProxy` in front of cockroachDB nodes. Everything works great until I kill off one of the nodes (which should be fine since the data is replicated in at least 2 of the 3 nodes) and we get a “successful” query returned with no valid data (even though there is data). I would expect `pgx` to return an error in the query to alert that something went wrong (trying to query a dead node). The next query gets rerouted by `HAProxy` to another node, we get an error that the database isn't connected, so we reconnect and get good data.

To replicate:

1. haproxy has connected program to cockroach node 1
1. Transactions are all good
1. Kill or stop cockroach node 1
1. haproxy recognizes cockroach node 1 is down
1. Program requests data via haproxy
1. Query returns without error but with zero results (even though there should be data)
1. Next query fails due to database connection error
1. Reconnection is successful as haproxy routes to another node
1. Good data is received

My main problem is that step number 6. No errors on a query that returns zero data when there is in fact matching data…I can’t figure out how to catch this failure mode and am hoping I’m missing some configuration somewhere? Or is this behavior expected? I can't throw an error on a query returning zero data as in production, that will be entirely possible.

Worth noting that when I connect directly to one of the cockroachdb nodes (and don’t use a proxy), when I kill the connected node, I immediately get an error (but then don’t have a load balancer to route to a different node).

Any help/insight is greatly appreciated. Thank you!

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by reproducing the HAProxy and CockroachDB failover sequence, then inspect pgx query and connection handling around the zero-result response and subsequent connection error. Done means establishing whether this behavior is expected and identifying the relevant configuration or implementation change.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.