openkruise / openkruise/agents-api
Feature request: provide io.ReadCloser-based streaming file read API in the Go SDK
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2
- Forks
- 7
- Avg merge
- 19h 24m
- Merged PRs (30d)
- 2
Description
Current behavior
The Go SDK only exposes Filesystem.Read / Filesystem.ReadText, both of which materialize the entire file in memory. There is no streaming entry point at all.
// runtime/filesystem.go:78-110
func (f *Filesystem) Read(ctx context.Context, path string, user ...string) ([]byte, error) {
...
return io.ReadAll(resp.Body) // whole file into heap
}
// runtime/filesystem.go:114-120
func (f *Filesystem) ReadText(ctx context.Context, path string, user ...string) (string, error) {
data, err := f.Read(ctx, path, user...) // and then copied again into a string
...
}
Peak memory is therefore file size for Read, and roughly twice the file size for ReadText.
This is the Go counterpart of #41 (Java), which was addressed by #45.
Additional problem specific to Go: a hard cap on total read time
Unlike the Java SDK — which has a dedicated streaming client (RuntimeConfig.getOrCreateStreamingHttpClient(), readTimeout=0) — the Go SDK has a single shared HTTP client:
// runtime/config.go:216 (defaultRequestTimeout = 60 * time.Second, runtime/config.go:16)
c.httpClient = &http.Client{Timeout: c.RequestTimeout}
In Go, http.Client.Timeout is a total deadline that includes reading the response body. Consequences:
- Even today,
Readcannot fetch a file whose transfer takes longer than 60s, regardless of available memory. - A naive
ReadStreamthat reuses this client would have its stream torn down at 60s — the same class of defect that was caught in the first review round of #45, but harder to notice in Go because there is no ready-made streaming client to pick from.
Desired behavior
Add a streaming read that returns io.ReadCloser, and back it with a client that does not impose a total timeout, letting the caller's ctx govern cancellation (all Filesystem methods already take a ctx).
Scope: only one place to change
Unlike the Java SDK, which keeps two hand-maintained copies of Filesystem.java, the Go side has a single implementation. e2b.Sandbox embeds *runtime.Client:
// e2b/sandbox.go:111-112
type Sandbox struct {
*runtime.Client
...
}
So adding ReadStream to runtime/filesystem.go automatically makes it available to both the standalone runtime client and the e2b client (sb.Files.ReadStream(...)). No synchronization work is needed.
Suggested API
// ReadStream opens a stream to the file content. The caller must Close the
// returned ReadCloser. Cancellation and deadlines are controlled via ctx.
func (f *Filesystem) ReadStream(ctx context.Context, path string, user ...string) (io.ReadCloser, error)
Usage:
rc, err := sb.Files.ReadStream(ctx, "/tmp/large.log")
if err != nil {
return err
}
defer rc.Close()
sc := bufio.NewScanner(rc)
for sc.Scan() {
process(sc.Text())
}
return sc.Err()
Implementation notes:
- Return
resp.Bodydirectly. Go needs no equivalent of the JavaFilterInputStreamwrapper — closing the body already releases the connection. - Add a streaming client in
ConfigwithTimeout: 0, reusing the sameTransport/connection pool. When a custom client was supplied viaWithHTTPClient, shallow-copy it and zero outTimeout(mirroring Java'snewBuilder().readTimeout(0)). - A
ReadTextStreamcounterpart is not proposed:bufio.NewScanner(rc)already covers it, and adding it would be un-idiomatic in Go. Readmay share an internal helper withReadStreamto avoid duplicating URL/request construction, but it should keep using the existing finite-timeout client so its behavior does not change.- Documentation to update: the filesystem API tables in
e2b/README.md:357,runtime/README.md:204, and theirREADME_zh-CH.mdcounterparts.
Use case
Same as #41: sandbox workloads produce logs, dumps and other large artifacts. Fetching them through Read/ReadText is impractical both because of heap usage and because of the 60s total-request deadline.
Separate follow-up, out of scope here: the same 60s client is also shared with the two Connect RPC clients (runtime/client.go:39-40), and long-lived server streams (runtime/command_handle.go:97, runtime/commands.go:173) run over it. Command output streams lasting more than RequestTimeout are likely to be truncated for the same reason. This needs to be confirmed and tracked in its own issue.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with runtime/filesystem.go and runtime/config.go, then inspect the HTTP client setup and the Filesystem methods around the lines cited in the issue. Confirm the streaming entry point uses the configured context for cancellation without a total timeout, preserves existing Read behavior, is exposed through e2b.Sandbox, and update the API tables in e2b/README.md, runtime/README.md, and their Chinese counterparts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, documentation
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100