aws / aws/aws-dotnet-distributed-cache-provider

Item size has exceeded the maximum allowed size

Open
#42 4 comments 0 reactions 0 assignees View on GitHub
feature-request m p2 queued
Dominant language
C#
Stars
47
Forks
3
Avg merge
1d 11h
Merged PRs (30d)
6

Description

### Describe the bug

When you try to save an item larger than 400kb the internal exception occurs:

`Failed to put item with key xxxx Caused by The item size has exceeded the maximum allowed size.`

### Expected Behavior

Compress the byte array using gzip or split the payload and save the item

### Current Behavior

An internal DynamoDB exception is throwed.

### Reproduction Steps

To reproduce this error, follow the steps below:

1- Perform the integration of the library
2- Try to save a list with multiple items over 400kb.

### Possible Solution

I actually use this extensions method class that allow save large payloads:

```csharp
public static class IDistributedCacheExtensions
{
private static readonly JsonSerializerOptions JsonSerializerOptions = new JsonSerializerOptions
{
PropertyNameCaseInsensitive = true,
};

public static async Task SetAsync(this IDistributedCache distributedCache, string key, T value, DistributedCacheEntryOptions options, CancellationToken token = default(CancellationToken))
{
var serializedValue = Serialize(value);
var compressedValue = Compress(serializedValue);

await distributedCache.SetAsync(key, compressedValue, options);
}

public static async Task GetAsync(this IDistributedCache distributedCache, string key, CancellationToken token = default(CancellationToken))
where T : class
{
var compressedValue = await distributedCache.GetAsync(key);

if (compressedValue != null)
{
var value = Decompress(compressedValue);
return Deserialize(value);
}

return default(T);
}

private static T Deserialize(byte[] bytes)
{
if (bytes == null)
{
return default(T);
}

return JsonSerializer.Deserialize(bytes, JsonSerializerOptions);
}

private static byte[] Serialize(T obj)
{
if (obj == null)
{
return null;
}

return JsonSerializer.SerializeToUtf8Bytes(obj, JsonSerializerOptions);
}

private static byte[] Compress(byte[] bytes)
{
using (var outputStream = new MemoryStream())
{
using (var gzipStream = new GZipStream(outputStream, CompressionMode.Compress))
{
gzipStream.Write(bytes, 0, bytes.Length);
}

return outputStream.ToArray();
}
}

private static byte[] Decompress(byte[] compressedBytes)
{
using (var compressedStream = new MemoryStream(compressedBytes))
{
using (var decompressedStream = new MemoryStream())
{
using (var gzipStream = new GZipStream(compressedStream, CompressionMode.Decompress))
{
gzipStream.CopyTo(decompressedStream);
}

return decompressedStream.ToArray();
}
}
}
}

```

### Additional Information/Context

_No response_

### AWS.AspNetCore.DistributedCacheProvider package version

AWS.AspNetCore.DistributedCacheProvider 0.9.0

### Targeted .NET Platform

NET 6

### Operating System and version

linux

Contributor guide

Open the contributing guide

Research direction

Start by locating the DynamoDB item-write path in the AWS.AspNetCore.DistributedCacheProvider package and reproduce the failure with a payload over 400 KB on .NET 6. Compare the possible compression and payload-splitting approaches described in the issue; completion criteria are not settled because the issue does not choose between them or name tests and files.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, csharp
Domain
backend, cloud, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.