googleapis / googleapis/google-cloud-dotnet

Multipart Upload with GCS .NET SDK

Open
#13,021 7 comments 0 reactions 1 assignee Claimed by @rajeevpodar View on GitHub
api: storage type: feature request
Dominant language
C#
Stars
1.1k
Forks
411
Avg merge
7h 29m
Merged PRs (30d)
57

Description

Hey guys, I recently developed a proof of concept about on-demand csv files generation.
The goal is to retrieve data from a relational database, map to csv and then upload it to bucket on cloud given a chunksize (eg 5MB).
I've read the [docs](https://cloud.google.com/storage/docs/resumable-uploads#unknown-resumables) and tried to use Resumable Upload feature, but my file gets overwritten.

A "complete" method would be very useful...but i didn't find anything about it. Could you help out here?

Successful sample code with AWS provider:
```c#
using System.Buffers;
using System.Globalization;
using System.IO;

using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;

using CsvHelper;

using Report.Generator.Infra.Repositories;

namespace Report.Generator.Infra.Generators
{
public class CsvAWSReportGenerator
{
private readonly IAmazonS3 _s3Client;

public CsvAWSReportGenerator(IAmazonS3 s3Client)
{
_s3Client = s3Client;
}

public async Task GenerateAsync(string bucketName, string keyName, int blockSize)
{
byte[] buffer = ArrayPool.Shared.Rent(blockSize);
int bufferPosition = 0;

List uploadResponses = new();
InitiateMultipartUploadRequest initiateRequest = new()
{
BucketName = bucketName,
Key = keyName
};

InitiateMultipartUploadResponse initResponse =
await _s3Client.InitiateMultipartUploadAsync(initiateRequest);

try
{
using var memoryStream = new MemoryStream();
using var writer = new StreamWriter(memoryStream);
using var csvWriter = new CsvWriter(writer, CultureInfo.InvariantCulture);

int partNumber = 1;

await foreach (var product in ProductRepository.FetchProductsAsync())
{
memoryStream.SetLength(0);
csvWriter.WriteRecord(product);
await csvWriter.NextRecordAsync();
await writer.FlushAsync();
memoryStream.Position = 0;

while (memoryStream.Position < memoryStream.Length)
{
int bytesToRead = Math.Min(blockSize - bufferPosition, (int)(memoryStream.Length - memoryStream.Position));
int bytesRead = await memoryStream.ReadAsync(buffer, bufferPosition, bytesToRead);
bufferPosition += bytesRead;

if (bufferPosition == blockSize)
{
await UploadPartAsync(buffer, bufferPosition, bucketName, keyName, initResponse.UploadId, partNumber++, uploadResponses);
bufferPosition = 0;
}
}
}

if (bufferPosition > 0)
{
await UploadPartAsync(buffer, bufferPosition, bucketName, keyName, initResponse.UploadId, partNumber, uploadResponses);
}

ArrayPool.Shared.Return(buffer);

CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest
{
BucketName = bucketName,
Key = keyName,
UploadId = initResponse.UploadId
};

completeRequest.AddPartETags(uploadResponses);

CompleteMultipartUploadResponse completeUploadResponse =
await _s3Client.CompleteMultipartUploadAsync(completeRequest);
}
catch (Exception exception)
{
Console.WriteLine("An AmazonS3Exception was thrown: {0}", exception.Message);

AbortMultipartUploadRequest abortMPURequest = new AbortMultipartUploadRequest
{
BucketName = bucketName,
Key = keyName,
UploadId = initResponse.UploadId
};
await _s3Client.AbortMultipartUploadAsync(abortMPURequest);
}
}

private async Task UploadPartAsync(byte[] buffer, int bufferLength, string bucketName, string keyName, string uploadId, int partNumber, List uploadResponses)
{
using var partStream = new MemoryStream(buffer, 0, bufferLength);
UploadPartRequest uploadRequest = new UploadPartRequest
{
BucketName = bucketName,
Key = keyName,
UploadId = uploadId,
PartNumber = partNumber,
PartSize = bufferLength,
InputStream = partStream
};

uploadRequest.StreamTransferProgress += new EventHandler(UploadPartProgressEventCallback);
uploadResponses.Add(await _s3Client.UploadPartAsync(uploadRequest));
}

public void UploadPartProgressEventCallback(object sender, StreamTransferProgressArgs e)
{
Console.WriteLine("{0}/{1}", e.TransferredBytes, e.TotalBytes);
}
}
}
```

PS: I've seen GCS does have a [XML multipart upload API](https://cloud.google.com/storage/docs/xml-api/post-object-complete).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.