Lambda support for DynamoDb Kinesis Stream processing
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 503
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 18
Description
### Describe the feature
Given a DynamoDb table configured with a Kinesis Stream and a Lambda to process the Kinesis Stream, the `Amazon.Lambda.KinesisEvents` package provides `KinesisEvent` as the lambda input object with the following structure
```cs
public class KinesisEvent
{
public IList Records { get; set; }
public class Record : Amazon.Kinesis.Model.Record
{
public string KinesisSchemaVersion { get; set; }
}
public class KinesisEventRecord
{
/// A bunch of other properties
...
public KinesisEvent.Record Kinesis { get; set; }
}
}
public class Record
{
/// A bunch of other properties
...
public MemoryStream Data { get; set; }
}
```
In the case of a DynamoDb Kinesis Stream the `Record.Data` `MemoryStream` contains a non-encoded Json string which appears to match the type `Amazon.DynamoDBv2.Model.Record` from the `AWSSDK.DynamoDBv2` package. Unfortunately the Json cannot be easily deserialized into a `Amazon.DynamoDBv2.Model.Record` due to the nested property `ApproximateCreationDateTime` which is in the form of epoch time in the JSON (i.e. an integer) but is typed as `DateTime` in `Amazon.DynamoDBv2.Model.StreamRecord`. I'm also concerned that there might be breaking edge-cases when directly deserializing `AttributeValue` types which are also nested inside `Amazon.DynamoDBv2.Model.StreamRecord`.
I seem to be able to deserialize `Record.Data` successfully using the `Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer` serializer which is really not intuitive and disappointing to have to do.
I'm wondering if it would be reasonable to either create a `KinesisDynamoDbEvent` type either in `Amazon.Lambda.KinesisEvents` or another package which types the nested `Data` property as a DynamoDb `StreamRecord` instead of a MemoryStream, or perhaps provide an intuitive utility to deserialize the `MemoryStream` appropriately.
### Use Case
When using Lambda to process a Kinesis Stream that is backed by a DynamoDb table it is difficult to deserialize the `Amazon.Kinesis.Model.Record` `MemoryStream` property to a dynamodb record.
### Proposed Solution
_No response_
### Other Information
_No response_
### Acknowledgements
- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### AWS .NET SDK and/or Package version used
Amazon.Lambda.KinesisEvents
### Targeted .NET Platform
.NET 6 and up
### Operating System and version
N/A
Contributor guide
Assessment
This issue has not been assessed yet.