aws / aws/aws-dotnet-distributed-cache-provider
Item size has exceeded the maximum allowed size
- Dominant language
- C#
- Stars
- 47
- Forks
- 3
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 6
Description
### Describe the bug
When you try to save an item larger than 400kb the internal exception occurs:
`Failed to put item with key xxxx Caused by The item size has exceeded the maximum allowed size.`
### Expected Behavior
Compress the byte array using gzip or split the payload and save the item
### Current Behavior
An internal DynamoDB exception is throwed.
### Reproduction Steps
To reproduce this error, follow the steps below:
1- Perform the integration of the library
2- Try to save a list with multiple items over 400kb.
### Possible Solution
I actually use this extensions method class that allow save large payloads:
```csharp
public static class IDistributedCacheExtensions
{
private static readonly JsonSerializerOptions JsonSerializerOptions = new JsonSerializerOptions
{
PropertyNameCaseInsensitive = true,
};
public static async Task SetAsync(this IDistributedCache distributedCache, string key, T value, DistributedCacheEntryOptions options, CancellationToken token = default(CancellationToken))
{
var serializedValue = Serialize(value);
var compressedValue = Compress(serializedValue);
await distributedCache.SetAsync(key, compressedValue, options);
}
public static async Task GetAsync(this IDistributedCache distributedCache, string key, CancellationToken token = default(CancellationToken))
where T : class
{
var compressedValue = await distributedCache.GetAsync(key);
if (compressedValue != null)
{
var value = Decompress(compressedValue);
return Deserialize(value);
}
return default(T);
}
private static T Deserialize(byte[] bytes)
{
if (bytes == null)
{
return default(T);
}
return JsonSerializer.Deserialize(bytes, JsonSerializerOptions);
}
private static byte[] Serialize(T obj)
{
if (obj == null)
{
return null;
}
return JsonSerializer.SerializeToUtf8Bytes(obj, JsonSerializerOptions);
}
private static byte[] Compress(byte[] bytes)
{
using (var outputStream = new MemoryStream())
{
using (var gzipStream = new GZipStream(outputStream, CompressionMode.Compress))
{
gzipStream.Write(bytes, 0, bytes.Length);
}
return outputStream.ToArray();
}
}
private static byte[] Decompress(byte[] compressedBytes)
{
using (var compressedStream = new MemoryStream(compressedBytes))
{
using (var decompressedStream = new MemoryStream())
{
using (var gzipStream = new GZipStream(compressedStream, CompressionMode.Decompress))
{
gzipStream.CopyTo(decompressedStream);
}
return decompressedStream.ToArray();
}
}
}
}
```
### Additional Information/Context
_No response_
### AWS.AspNetCore.DistributedCacheProvider package version
AWS.AspNetCore.DistributedCacheProvider 0.9.0
### Targeted .NET Platform
NET 6
### Operating System and version
linux
Contributor guide
Research direction
Start by locating the DynamoDB item-write path in the AWS.AspNetCore.DistributedCacheProvider package and reproduce the failure with a payload over 400 KB on .NET 6. Compare the possible compression and payload-splitting approaches described in the issue; completion criteria are not settled because the issue does not choose between them or name tests and files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, csharp
- Domain
- backend, cloud, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100