aws / aws/aws-lambda-dotnet

Lambda support for DynamoDb Kinesis Stream processing

Open
#1,554 6 comments 1 reaction 0 assignees View on GitHub
feature-request module/lambda-client-lib p2 queued
Dominant language
C#
Stars
1.7k
Forks
503
Avg merge
1d 5h
Merged PRs (30d)
18

Description

### Describe the feature

Given a DynamoDb table configured with a Kinesis Stream and a Lambda to process the Kinesis Stream, the `Amazon.Lambda.KinesisEvents` package provides `KinesisEvent` as the lambda input object with the following structure
```cs
public class KinesisEvent
{
public IList Records { get; set; }

public class Record : Amazon.Kinesis.Model.Record
{
public string KinesisSchemaVersion { get; set; }
}

public class KinesisEventRecord
{
/// A bunch of other properties
...

public KinesisEvent.Record Kinesis { get; set; }
}
}

public class Record
{
/// A bunch of other properties
...

public MemoryStream Data { get; set; }
}
```

In the case of a DynamoDb Kinesis Stream the `Record.Data` `MemoryStream` contains a non-encoded Json string which appears to match the type `Amazon.DynamoDBv2.Model.Record` from the `AWSSDK.DynamoDBv2` package. Unfortunately the Json cannot be easily deserialized into a `Amazon.DynamoDBv2.Model.Record` due to the nested property `ApproximateCreationDateTime` which is in the form of epoch time in the JSON (i.e. an integer) but is typed as `DateTime` in `Amazon.DynamoDBv2.Model.StreamRecord`. I'm also concerned that there might be breaking edge-cases when directly deserializing `AttributeValue` types which are also nested inside `Amazon.DynamoDBv2.Model.StreamRecord`.

I seem to be able to deserialize `Record.Data` successfully using the `Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer` serializer which is really not intuitive and disappointing to have to do.

I'm wondering if it would be reasonable to either create a `KinesisDynamoDbEvent` type either in `Amazon.Lambda.KinesisEvents` or another package which types the nested `Data` property as a DynamoDb `StreamRecord` instead of a MemoryStream, or perhaps provide an intuitive utility to deserialize the `MemoryStream` appropriately.

### Use Case

When using Lambda to process a Kinesis Stream that is backed by a DynamoDb table it is difficult to deserialize the `Amazon.Kinesis.Model.Record` `MemoryStream` property to a dynamodb record.

### Proposed Solution

_No response_

### Other Information

_No response_

### Acknowledgements

- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### AWS .NET SDK and/or Package version used

Amazon.Lambda.KinesisEvents

### Targeted .NET Platform

.NET 6 and up

### Operating System and version

N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.