apple / apple/foundationdb

Some kind of socket/file handle leak when using many BackupContainerBlobStore instances

Open
#2,433 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

I'm not certain of the exact nature or scope of this issue, but here is what I know:

On MacOS 10.15 (and possibly other OSs, I don't know), the following sequence will eventually result in an inability to connect to any hosts referenced by any BlobStoreEndpoint URL:

1) Open a Reference via openContainer() with a blobstore:// URL
2) Call and wait for ->exists() on the container.
3) Drop the reference
4) Repeat

Eventually, connection attempts will fail, with either connection_failed or dns_lookup_failed. At this point, it is *also* impossible to open any new files, as there are no available file handles. So something about this sequence seems to leak a socket handle.

Each container instance creates its own BlobStoreEndpoint instance and each of those maintain their own separate connection pool, so step 2 above is resolving the hostname, picking an IP, and opening a connection each time. However in step 3 the backup container is destroyed along with its blob store endpoint and then its connection pool which holds all of the network connection handles, so there should only be one connection in existence at any one time.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the sequence through openContainer() and exists() with repeated BackupContainerBlobStore instances on macOS. Trace destruction of the BlobStoreEndpoint and its connection pool after each container is dropped, and verify that connections and file handles are released so later host connections and file opens continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.