redpanda-data / redpanda-data/connect
Support Batching in the Azure Blob Storage output
@mihaitodor is already working on this.
Since Sep 23, 2021.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
There's (probably) no real reason why it couldn't support batching just as well as the S3 output does.
Godocs for uploading a Block Blob: https://pkg.go.dev/github.com/Azure/azure-storage-blob-go/azblob#BlockBlobURL.Upload
It mentions that you can't update existing Block Blobs, and that you can stage Blobs before you commit them to the final Block Blob.
To perform a partial update of a block blob, use StageBlock and CommitBlockList
This could probably be used to upload large files more efficiently? Not sure it's worth it since Benthos is probably more likely to pass many small to medium sized payloads rather than few large ones.
Microsoft docs for Put Blob: https://docs.microsoft.com/en-gb/rest/api/storageservices/put-blob
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.