keybase / keybase/client

kbfsfuse: memory consumption and cpu usage keeps increasing during `keybase fs write`

Open
#24,867 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
9.2k
Forks
1.3k
Avg merge
12h 58m
Merged PRs (30d)
56

Description

Hello. I'm using `keybase fs write` to push real-time-generated data to KBFS. The rate of the data itself is quite low, about 16 kB/s, but it's a long running process, and I've noticed that as long as `keybase fs write` is running, `kbfsfuse` will slowly but steadily consume increasing amounts of memory and CPU time.
I was able to come up with a minimal reproducible example, published here: https://gist.github.com/jpovixwm/b75beed3266a84c06b85b743cbe8b89d
It uses a bunch of additional switches that I've added in an attempt to keep the resource consumption to a minimum. But since I'm not a Keybase expert, it could very well be the case that it's the combination of these switches that is causing the memory leak.
Even so, I think it should not be possible to create this kind of conditions via the use of command line options that are exposed to the user.
I've used `GOGC=1` and `GODEBUG=madvdontneed=1` env vars to make the Go GC run more often and to have it free unused memory, so that the container's memory consumption better visualizes actual consumption (hopefully).
![image](https://user-images.githubusercontent.com/53523617/159115796-7536aeb8-2d8b-49fa-9637-b64c6db1ff77.png)

my log id: 24fe061d92b7bad6accfa11c

Edit: after some additional time, I can see the CPU usage doesn't increase any further, while memory consumption continues to go up:
![image](https://user-images.githubusercontent.com/53523617/159190466-1acf5805-53dc-4f20-9825-904a70cdcdea.png)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the minimal reproducible example linked in the issue with `keybase fs write` and observe `kbfsfuse` memory and CPU usage over time. Use the reported log id and the listed `GOGC` and `GODEBUG` settings when comparing results; done means the long-running write no longer causes steadily increasing memory consumption.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.