apache / apache/dolphinscheduler
[Improvement] [API] Apiserver OOM when downloading large task log
- Dominant language
- Java
- Stars
- 14.5k
- Forks
- 5.1k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 29
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
When a user downloads a large task log, the apiserver can run out of memory. The whole RPC response body is read into memory at once. In our heap dump we saw a `TransporterDecoder` whose `body` was a `byte[]` of ~226MB — it was a `queryTaskResult` response. On 3.3.2 the decoder does `new byte[bodyLength]` without any size check, so any large response gets allocated whole.
Also, once a `maxFrameSize` guard is in place, a task log larger than the frame limit can no longer be downloaded — the log is returned as one big `byte[]` in a single RPC, so the request is rejected.
### What you expected to happen
Large task logs should still be downloadable (streamed in chunks, not one big RPC), and the server shouldn't OOM on a large response.
### How to reproduce
1. Run a task that produces a large log file.
2. Click "Download full log" or view the task result.
3. On 3.3.2 the apiserver OOMs; with the frame guard, the large-log download fails.
### Anything else
The OOM side is handled by the `maxFrameSize` check in `TransporterDecoder`. This issue also tracks making large-log download work under that limit (stream the log in chunks).
### Version
3.3.2
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.