googleapis / googleapis/google-cloud-go

storage: Have Writer implement io.ReaderFrom

Open
#8,622 0 comments 0 reactions 1 assignee Assigned to @cpriti-os View on GitHub
api: storage priority: p3 type: feature request
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

Feature Request:

I have an `io.Reader` of data that I want to write to cloud storage using the google-cloud-go client library.

`b.bucket.Object(path).NewWriter(ctx)` returns an `io.Writer`

To copy the reader's data into the writer, I have to copy to an intermediate buffer, such as:

```go
writer := b.bucket.Object(path).NewWriter(ctx)
io.Copy(writer, reader)
```

(https://github.com/golang/go/blob/6cf6067d4eb20dfb3d31c0a8ccdbfdf0bf304b72/src/io/io.go#L387)

However, as an optimization, I would like to avoid this intermediary buffer and copy.
The `io.Copy` function checks if the writer implements `ReaderFrom`, and if it does, the intermediate allocation can be skipped.
See: https://github.com/golang/go/blob/6cf6067d4eb20dfb3d31c0a8ccdbfdf0bf304b72/src/io/io.go#L414

For all internally implemented `io.Reader`'s, I can also implement the `io.WriterTo` interface, to optimize this path. But, some of my reader's are from external sources that I cannot control.

This would be both a performance and UX improvement. AWS's S3 SDK and Azure's BlobStorage SDK for Go both use the pattern of accepting an `io.Reader` for writing data, rather than returning an `io.Writer` to use.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.