aws / aws/aws-lambda-go

[BUG] Incorrect unmarshalling self-hosted Kafka event

Open
#422 1 comment 0 reactions 1 assignee Claimed by @bmoffatt View on GitHub
bug type/events
Dominant language
Go
Stars
3.8k
Forks
578
Avg merge
8h 18m
Merged PRs (30d)
1

Description

We have a lambda function which is hooked up and listens to both MSK and self-hosted Kafka instances to process incoming messages. While in case is MSK the event source mapping works fine, for self-hosted event source mapping SDK panics with error as follow:
```json
{
"errorMessage": "json: cannot unmarshal number -23 into Go struct field KafkaRecord.records.headers of type uint8",
"errorType": "UnmarshalTypeError"
}
```

This seems like some type mismatch (`uint8`) so it can't properly unmarshal negative numbers coming in the lambda event payload.

We had opened AWS support case and worked with them to some degree but unfortunately they couldn't provide us much context so we did debugging ourselves.

The issue seems to be present in the versions `1.26+` of the Lambda SDK, and downgrading to 1.25 or below doesn't cause this unmarshal error to appear. The main different is the presence of the `headers` field in the struct introduced through https://github.com/aws/aws-lambda-go/pull/385/files#diff-cf0e5ac9714618f1821f8694da87d7af4579e0eafed210e8dbdcc5977bf5a69eR20. This might also be related to some previous issue https://github.com/aws/aws-lambda-go/issues/390.

As mentioned above this only applies to self-hosted event mapping, not MSK events.

Environment:

- Go: `1.17`
- AWS LAmbda SDK Go: `1.26`, `1.27`, `1.28`
- Kafka version (self-hosted): `2.7.0`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.