hashicorp / hashicorp/boundary
Performance issues due to untidy session_connection table
- Dominant language
- Go
- Stars
- 4.1k
- Forks
- 317
- Avg merge
- 5h 10m
- Merged PRs (30d)
- 1
Description
**Describe the bug**
For now very little is known about the problem. `psql` sessions break with
```
SSL SYSCALL error: EOF detected
```
The symptom only arises when a certain data pipeline runs 1000s of tests using `dbt` against a Postgres instance proxied via Boundary. It uses high concurrency (8 parallel jobs) and each job can create multiple session connections because of the way the underlying `libpq` parallelizes the queries, so connection churn may be a factor.
Another thing we noticed, when that happens, it affects the other sessions on the same worker too, and as far as I can tell, it may affect sessions of users connected to other workers (i.e. in different clusters) - it's hard to say for sure, but it seems that the dropped connections are all related (which makes me think that something is affecting the controllers).
Other than that, sessions and connections work just fine.
We're running Boundary version `0.12.2` on both controllers and workers, as Kubernetes StatefulSets.
**To Reproduce**
At a high level:
1. Connect to a database target
2. Run `dbt` with 1000s of tests, with high concurrency, against a single target
3. Connection breaks after 30s-1m and also breaks other connections on the same worker. Allegedly even connections on other workers are affected.
**Expected behavior**
Stable connection even under heavy load and concurrency.
**Additional context**
We've have manually applied the fixes https://github.com/hashicorp/boundary/pull/3283 and https://github.com/hashicorp/boundary/pull/3280
I'm attaching logs from the workers where the machine that runs the data pipeline connects ([data-workers.txt](https://github.com/hashicorp/boundary/files/11706423/data-workers.txt)) and from the controllers and workers running on a different cluster ([controllers+workers.txt](https://github.com/hashicorp/boundary/files/11706422/controllers%2Bworkers.txt)). We noticed connection breakage on the workers of this cluster as well. The logs are JSON formatted (but extension was changed because Github didn't like `.json`) and span a 2-min interval in which we observed the issue. They are ordered from the most recent to the oldest.
I'd be happy to investigate the issue further and run any debugging commands or gather extra logs.
Thanks.
Contributor guide
Research direction
Start with the attached worker and controller logs and compare the behavior with fixes in PRs #3283 and #3280. Reproduce the high-concurrency dbt workload through Boundary against PostgreSQL, then narrow down why session connections are dropped across workers; done means stable connections under the stated load.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100