OpenListTeam / OpenListTeam/OpenList
[BUG] Memory leak during web uploads (Stream method): Go live heap grows unbounded, container gets OOM-killed
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 24.7k
- Forks
- 2.3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 36
Description
Please confirm the following
-
I have read and agree to AGPL-3.0 Section 15 .
The program is provided "as is" without any warranties; you bear all risks of using it. -
I have read and agree to AGPL-3.0 Section 16 .
The copyright holders and distributors are not liable for any damages resulting from the use or inability to use the program. -
I confirm my description is clear, polite, helps developers quickly locate the issue, and complies with community rules.
-
I have read the OpenList documentation.
-
I confirm there are no duplicate issues or discussions.
-
I confirm this is an
OpenListissue, not caused by other reasons (such as network, dependencies, or operation). -
I believe this issue must be handled by
OpenListand not by a third party. -
I confirm this issue is not fixed in the latest version.
-
I have not read these checkboxes and therefore I just ticked them all, Please close this issue.
OpenList Version (required)
4.2.6
Storage Driver Used (required)
Teldrive
Bug Description (required)
During a multi-file upload from the web UI the Go LIVE heap grows monotonically and is never
freed while uploads are in flight. GC cannot reclaim the memory (objects stay referenced),
runs in a death spiral near GOMEMLIMIT, and the process eventually exceeds its container
memory limit and is killed by the kernel (cgroup OOM). Consequences: all sessions invalidated
(server-side token cache is in-memory), reverse proxy returns 502, upload tasks are dropped.
Key facts:
- Grows with the NUMBER of parallel upload streams, not with file size: ~15 files x ~200 KB
(about 3 MB of data total) pushed the live heap from ~437 MB to 557+ MB. - Persists with chunk_size=2 MiB and upload_concurrency=1, so it is not driver chunk buffering.
- Memory is released instantly (RSS drops to ~68 MB) the moment the upload batch finishes —
the memory is referenced only while uploads are in flight. - GOMEMLIMIT=380MiB cannot help: GC runs continuously but the heap is live, not garbage.
- The built-in backpressure (min_free_memory / auto_memory_limit) reads host-wide free memory
(/proc/meminfo), which is wrong when OpenList itself runs in a container with its own limit —
it never throttles even though the container is about to be killed.
Expected behavior: upload pipeline should apply backpressure based on its own cgroup memory
usage (e.g. /sys/fs/cgroup/memory.current) and/or stop retaining references to completed
upload state. Ideally: pause accepting new chunks/streams when the process approaches its
memory limit instead of being OOM-killed.
Logs (required)
GODEBUG=gctrace=1 during a Stream upload (~15 files, ~200 KB each, ~3 MB total):
gc 92 @107.4s 437->437->437 MB, 437 MB goal
gc 130 @109.6s 488->488->488 MB
gc 160 @111.4s 532->536->536 MB
gc 171 @112.1s 554->557->554 MB
Hundreds of GC cycles within seconds (death spiral), live heap (last number) keeps climbing.
Kernel side (cgroup limit 320 MiB at the time):
[12:52:28] oom-kill:constraint=...,task=openlist,uid=1001
[12:52:28] Out of memory: Killed process 67767 (openlist) total-vm:3923960kB, anon-rss:731308kB
[13:32:02] Memory cgroup out of memory: Killed process 74546 (openlist) anon-rss:301604kB
[13:40:20] Memory cgroup out of memory: Killed process 75528 (openlist) anon-rss:321912kB
After the batch finishes, RSS instantly drops to ~68 MB — nothing was freed by GC while uploading.
Configuration File Content (required)
config.json (sensitive fields masked):
{
"force": false,
"site_url": "",
"cdn": "",
"jwt_secret": "<masked>",
"token_expires_in": 48,
"database": { "type": "sqlite3", "db_file": "data/data.db", "table_prefix": "x_" },
"scheme": { "address": "0.0.0.0", "http_port": 5244 },
"temp_dir": "data/temp",
"log": { "enable": true, "name": "data/log/log.log" },
"auto_memory_limit": 200,
"min_free_memory": 120,
"max_concurrency": 64,
"s3": { "enable": false },
"ftp": { "enable": false },
"sftp": { "enable": false },
"mcp": { "enable": false },
"last_launched_version": "v4.2.6"
}
Teldrive storage driver addition (sensitive fields masked):
{
"root_folder_path": "/",
"url": "http://openlistfix:8080",
"cookie": "access_token=",
"use_share_link": false,
"chunk_size": 2,
"random_chunk_name": false,
"upload_concurrency": 1
}
AI Generated Content
- I used AI tools to generate this content
- I did not use AI tools to generate this content
AI model used
GLM 5.3 Flash
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Stream upload path and its upload pipeline, then reproduce the reported multi-file web upload using the supplied concurrency and memory settings. Compare live heap and cgroup memory during uploads and after completion to identify retained upload state and whether host-wide memory checks throttle the container. Done means in-flight uploads no longer cause unbounded live-heap growth or container OOM kills.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100