apple / apple/foundationdb

Go (golang bindings): 8 database connections at once causes CGO signal and panic

Open
#11,856 23 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

go version go1.23.3 linux/amd64

I'm given to understand from reading the FoundationDB documentation[1][2] that the key to getting acceptable write throughput is to write through many connections to the database in parallel.

However: when I try to do this, I saw increasing throughput going from 1 to 2, and from 2 to 4 connections;
but I get consistent panics of the "CGO received a signal" type when I try with 8 database connections.

See the crash logs linked below and the repo's kvfile/ directory for a reproducer (run the tests).

https://github.com/glycerine/fdb_demo_golang
https://github.com/glycerine/fdb_demo_golang/blob/master/crash.log.macos.txt
https://github.com/glycerine/fdb_demo_golang/blob/master/crash.log.txt
https://github.com/glycerine/fdb_demo_golang/blob/master/crash.log2.txt

run go test -v in the ./kvfile directory of the fdb_demo_golang repo to reproduce the panic.
The Test002 test in particular; you can change the database pool size here: https://github.com/glycerine/fdb_demo_golang/blob/master/kvfile/db_test.go#L97

Thanks,
Jason

@johscheuer because other Go issues here mention you (but apologies and please re-direct if this is not your area now)

references
[1] https://apple.github.io/foundationdb/features.html#concurrent-connections

"FoundationDB is able to handle large numbers of concurrent client connections. Because it uses a
threadless communications and concurrency model, FoundationDB does not have to create a
thread per connection. This allows full performance even with hundreds of thousands of in-flight
requests."

[2] https://apple.github.io/foundationdb/performance.html

"Its asynchronous design allows it to handle very high concurrency, and for a typical workload with
90% reads and 10% writes, maximum throughput is reached at about 200 concurrent operations.
This number of operations was achieved with 20 concurrent transactions per FoundationDB process
each running 10 operations with 16 byte keys and values between 8 and 100 bytes."

"The implication of this [Little's Law] relation is that, at a given latency, we can maximize throughput
only by concurrently submitting enough outstanding requests. A FoundationDB cluster might have a
commit latency of 2 ms and yet be capable of far more than 500 commits per second. In fact, tens of
thousands of commits per second are easily achievable. To achieve this rate, there must be hundreds
of requests happening concurrently. Not having enough pending requests is the single biggest
reason for low performance."

Contributor guide

Open the contributing guide

Research direction

Start with the reproducer's kvfile/db_test.go, especially Test002 and the database pool-size setting around line 97. Run go test -v in the kvfile directory and inspect the linked crash logs to characterize the CGO signal and panic at eight connections. Done means the reproducer no longer panics when the pool uses eight database connections.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.