kbfsfuse: memory consumption and cpu usage keeps increasing during `keybase fs write`
- Dominant language
- Go
- Stars
- 9.2k
- Forks
- 1.3k
- Avg merge
- 12h 58m
- Merged PRs (30d)
- 56
Description
Hello. I'm using `keybase fs write` to push real-time-generated data to KBFS. The rate of the data itself is quite low, about 16 kB/s, but it's a long running process, and I've noticed that as long as `keybase fs write` is running, `kbfsfuse` will slowly but steadily consume increasing amounts of memory and CPU time.
I was able to come up with a minimal reproducible example, published here: https://gist.github.com/jpovixwm/b75beed3266a84c06b85b743cbe8b89d
It uses a bunch of additional switches that I've added in an attempt to keep the resource consumption to a minimum. But since I'm not a Keybase expert, it could very well be the case that it's the combination of these switches that is causing the memory leak.
Even so, I think it should not be possible to create this kind of conditions via the use of command line options that are exposed to the user.
I've used `GOGC=1` and `GODEBUG=madvdontneed=1` env vars to make the Go GC run more often and to have it free unused memory, so that the container's memory consumption better visualizes actual consumption (hopefully).

my log id: 24fe061d92b7bad6accfa11c
Edit: after some additional time, I can see the CPU usage doesn't increase any further, while memory consumption continues to go up:

Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the minimal reproducible example linked in the issue with `keybase fs write` and observe `kbfsfuse` memory and CPU usage over time. Use the reported log id and the listed `GOGC` and `GODEBUG` settings when comparing results; done means the long-running write no longer causes steadily increasing memory consumption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100