googleapis / googleapis/google-cloud-go

spanner: reuse decoding buffers from row to row

Open
#4,049 2 comments 0 reactions 1 assignee Claimed by @rahul2393 View on GitHub
api: spanner priority: p3 type: feature request
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

The following statement is from a discussion:

It is nice that spanner package has Decoder interface, but we don't use it because it is hard to reuse buffers (needed for conversion) from row to row.
We have https://pkg.go.dev/go.chromium.org/luci/resultdb/internal/spanutil#Buffer which we allocate before a loop-over-rows and use for many rows (usage). Example:

```go
var b spanutil.Buffer
err := spanutil.Query(ctx, st, func(row *spanner.Row) error {
var key testVariantKey
if err := b.FromSpanner(row, &key.testID, &key.variantHash); err != nil {
return err
}
tvs[key] = struct{}{}
return nil
})
```

Here is a better one: https://chromium.googlesource.com/infra/luci/luci-go/+/HEAD/resultdb/internal/testresults/query.go#159
We compress some fields with zstd encoder, and reuse decoding buffer from row to row.
Compression is implemented here: https://chromium.googlesource.com/infra/luci/luci-go/+/HEAD/resultdb/internal/spanutil/compression.go

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.