apache / apache/dolphinscheduler

[Improvement] [API] Apiserver OOM when downloading large task log

Open
#18,459 2 comments 0 reactions 1 assignee Claimed by @xmg333 View on GitHub
backend improvement
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
29

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.

### What happened

When a user downloads a large task log, the apiserver can run out of memory. The whole RPC response body is read into memory at once. In our heap dump we saw a `TransporterDecoder` whose `body` was a `byte[]` of ~226MB — it was a `queryTaskResult` response. On 3.3.2 the decoder does `new byte[bodyLength]` without any size check, so any large response gets allocated whole.

Also, once a `maxFrameSize` guard is in place, a task log larger than the frame limit can no longer be downloaded — the log is returned as one big `byte[]` in a single RPC, so the request is rejected.

### What you expected to happen

Large task logs should still be downloadable (streamed in chunks, not one big RPC), and the server shouldn't OOM on a large response.

### How to reproduce

1. Run a task that produces a large log file.
2. Click "Download full log" or view the task result.
3. On 3.3.2 the apiserver OOMs; with the frame guard, the large-log download fails.

### Anything else

The OOM side is handled by the `maxFrameSize` check in `TransporterDecoder`. This issue also tracks making large-log download work under that limit (stream the log in chunks).

### Version

3.3.2

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.