googleapis / googleapis/google-cloud-dotnet
Multipart Upload with GCS .NET SDK
- Dominant language
- C#
- Stars
- 1.1k
- Forks
- 411
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 57
Description
Hey guys, I recently developed a proof of concept about on-demand csv files generation.
The goal is to retrieve data from a relational database, map to csv and then upload it to bucket on cloud given a chunksize (eg 5MB).
I've read the [docs](https://cloud.google.com/storage/docs/resumable-uploads#unknown-resumables) and tried to use Resumable Upload feature, but my file gets overwritten.
A "complete" method would be very useful...but i didn't find anything about it. Could you help out here?
Successful sample code with AWS provider:
```c#
using System.Buffers;
using System.Globalization;
using System.IO;
using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;
using CsvHelper;
using Report.Generator.Infra.Repositories;
namespace Report.Generator.Infra.Generators
{
public class CsvAWSReportGenerator
{
private readonly IAmazonS3 _s3Client;
public CsvAWSReportGenerator(IAmazonS3 s3Client)
{
_s3Client = s3Client;
}
public async Task GenerateAsync(string bucketName, string keyName, int blockSize)
{
byte[] buffer = ArrayPool.Shared.Rent(blockSize);
int bufferPosition = 0;
List uploadResponses = new();
InitiateMultipartUploadRequest initiateRequest = new()
{
BucketName = bucketName,
Key = keyName
};
InitiateMultipartUploadResponse initResponse =
await _s3Client.InitiateMultipartUploadAsync(initiateRequest);
try
{
using var memoryStream = new MemoryStream();
using var writer = new StreamWriter(memoryStream);
using var csvWriter = new CsvWriter(writer, CultureInfo.InvariantCulture);
int partNumber = 1;
await foreach (var product in ProductRepository.FetchProductsAsync())
{
memoryStream.SetLength(0);
csvWriter.WriteRecord(product);
await csvWriter.NextRecordAsync();
await writer.FlushAsync();
memoryStream.Position = 0;
while (memoryStream.Position < memoryStream.Length)
{
int bytesToRead = Math.Min(blockSize - bufferPosition, (int)(memoryStream.Length - memoryStream.Position));
int bytesRead = await memoryStream.ReadAsync(buffer, bufferPosition, bytesToRead);
bufferPosition += bytesRead;
if (bufferPosition == blockSize)
{
await UploadPartAsync(buffer, bufferPosition, bucketName, keyName, initResponse.UploadId, partNumber++, uploadResponses);
bufferPosition = 0;
}
}
}
if (bufferPosition > 0)
{
await UploadPartAsync(buffer, bufferPosition, bucketName, keyName, initResponse.UploadId, partNumber, uploadResponses);
}
ArrayPool.Shared.Return(buffer);
CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest
{
BucketName = bucketName,
Key = keyName,
UploadId = initResponse.UploadId
};
completeRequest.AddPartETags(uploadResponses);
CompleteMultipartUploadResponse completeUploadResponse =
await _s3Client.CompleteMultipartUploadAsync(completeRequest);
}
catch (Exception exception)
{
Console.WriteLine("An AmazonS3Exception was thrown: {0}", exception.Message);
AbortMultipartUploadRequest abortMPURequest = new AbortMultipartUploadRequest
{
BucketName = bucketName,
Key = keyName,
UploadId = initResponse.UploadId
};
await _s3Client.AbortMultipartUploadAsync(abortMPURequest);
}
}
private async Task UploadPartAsync(byte[] buffer, int bufferLength, string bucketName, string keyName, string uploadId, int partNumber, List uploadResponses)
{
using var partStream = new MemoryStream(buffer, 0, bufferLength);
UploadPartRequest uploadRequest = new UploadPartRequest
{
BucketName = bucketName,
Key = keyName,
UploadId = uploadId,
PartNumber = partNumber,
PartSize = bufferLength,
InputStream = partStream
};
uploadRequest.StreamTransferProgress += new EventHandler(UploadPartProgressEventCallback);
uploadResponses.Add(await _s3Client.UploadPartAsync(uploadRequest));
}
public void UploadPartProgressEventCallback(object sender, StreamTransferProgressArgs e)
{
Console.WriteLine("{0}/{1}", e.TransferredBytes, e.TotalBytes);
}
}
}
```
PS: I've seen GCS does have a [XML multipart upload API](https://cloud.google.com/storage/docs/xml-api/post-object-complete).
Contributor guide
Assessment
This issue has not been assessed yet.