openkruise / openkruise/agents-api

Feature request: provide io.ReadCloser-based streaming file read API in the Go SDK

Open
#46 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
2
Forks
7
Avg merge
19h 24m
Merged PRs (30d)
2

Description

Current behavior

The Go SDK only exposes Filesystem.Read / Filesystem.ReadText, both of which materialize the entire file in memory. There is no streaming entry point at all.

// runtime/filesystem.go:78-110
func (f *Filesystem) Read(ctx context.Context, path string, user ...string) ([]byte, error) {
    ...
    return io.ReadAll(resp.Body) // whole file into heap
}

// runtime/filesystem.go:114-120
func (f *Filesystem) ReadText(ctx context.Context, path string, user ...string) (string, error) {
    data, err := f.Read(ctx, path, user...)   // and then copied again into a string
    ...
}

Peak memory is therefore file size for Read, and roughly twice the file size for ReadText.

This is the Go counterpart of #41 (Java), which was addressed by #45.

Additional problem specific to Go: a hard cap on total read time

Unlike the Java SDK — which has a dedicated streaming client (RuntimeConfig.getOrCreateStreamingHttpClient(), readTimeout=0) — the Go SDK has a single shared HTTP client:

// runtime/config.go:216 (defaultRequestTimeout = 60 * time.Second, runtime/config.go:16)
c.httpClient = &http.Client{Timeout: c.RequestTimeout}

In Go, http.Client.Timeout is a total deadline that includes reading the response body. Consequences:

  1. Even today, Read cannot fetch a file whose transfer takes longer than 60s, regardless of available memory.
  2. A naive ReadStream that reuses this client would have its stream torn down at 60s — the same class of defect that was caught in the first review round of #45, but harder to notice in Go because there is no ready-made streaming client to pick from.

Desired behavior

Add a streaming read that returns io.ReadCloser, and back it with a client that does not impose a total timeout, letting the caller's ctx govern cancellation (all Filesystem methods already take a ctx).

Scope: only one place to change

Unlike the Java SDK, which keeps two hand-maintained copies of Filesystem.java, the Go side has a single implementation. e2b.Sandbox embeds *runtime.Client:

// e2b/sandbox.go:111-112
type Sandbox struct {
    *runtime.Client
    ...
}

So adding ReadStream to runtime/filesystem.go automatically makes it available to both the standalone runtime client and the e2b client (sb.Files.ReadStream(...)). No synchronization work is needed.

Suggested API

// ReadStream opens a stream to the file content. The caller must Close the
// returned ReadCloser. Cancellation and deadlines are controlled via ctx.
func (f *Filesystem) ReadStream(ctx context.Context, path string, user ...string) (io.ReadCloser, error)

Usage:

rc, err := sb.Files.ReadStream(ctx, "/tmp/large.log")
if err != nil {
    return err
}
defer rc.Close()

sc := bufio.NewScanner(rc)
for sc.Scan() {
    process(sc.Text())
}
return sc.Err()

Implementation notes:

  • Return resp.Body directly. Go needs no equivalent of the Java FilterInputStream wrapper — closing the body already releases the connection.
  • Add a streaming client in Config with Timeout: 0, reusing the same Transport/connection pool. When a custom client was supplied via WithHTTPClient, shallow-copy it and zero out Timeout (mirroring Java's newBuilder().readTimeout(0)).
  • A ReadTextStream counterpart is not proposed: bufio.NewScanner(rc) already covers it, and adding it would be un-idiomatic in Go.
  • Read may share an internal helper with ReadStream to avoid duplicating URL/request construction, but it should keep using the existing finite-timeout client so its behavior does not change.
  • Documentation to update: the filesystem API tables in e2b/README.md:357, runtime/README.md:204, and their README_zh-CH.md counterparts.

Use case

Same as #41: sandbox workloads produce logs, dumps and other large artifacts. Fetching them through Read/ReadText is impractical both because of heap usage and because of the 60s total-request deadline.


Separate follow-up, out of scope here: the same 60s client is also shared with the two Connect RPC clients (runtime/client.go:39-40), and long-lived server streams (runtime/command_handle.go:97, runtime/commands.go:173) run over it. Command output streams lasting more than RequestTimeout are likely to be truncated for the same reason. This needs to be confirmed and tracked in its own issue.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with runtime/filesystem.go and runtime/config.go, then inspect the HTTP client setup and the Filesystem methods around the lines cited in the issue. Confirm the streaming entry point uses the configured context for cancellation without a total timeout, preserves existing Read behavior, is exposed through e2b.Sandbox, and update the API tables in e2b/README.md, runtime/README.md, and their Chinese counterparts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.