aws / aws/aws-cdk

(aws.events): Double Quotes removed from the event which breaks the valid JSON object

Open
#15,512 4 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-events bug closed-for-staleness effort/medium p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

Hi,

I am trying to write incoming events from EventBus/EventBridge into Kinesis Firehose and the data is written into S3.
What I am trying to accomplish here, take the JSON data from the eventbus and write it as a newline into the S3. I have seen some suggestions to use Lambda function in the Firehose, but I think this should be technically possible (and is done on the UI)

I am not sure what is exactly the controlling party in the whole process that does the translation from the definition into the proper objects (since the text is extracted into InputTemplate and InputPath. My assumption is that `RuleTargetInput.fromMultilineText` is the culprit on the CDK.

The only way I got it working the way I wanted is using the AWS UI.

Input Transformer First Text box ->
```
{"detail":"$.detail"}
```
Input Transformer Second Text box (notice the empty line)->
```
{ "event": }

```
Since the above config done on the UI, I cannot tell what the CDK code would be like. (Also it got me quite a bit of time to find the right combination)

### What's happening

When the `RuleTargetInput.fromObject` method is used, the JSON object is appended next to the each other, thus it's impossible to add a newline.
When the `RuleTargetInput.fromMultilineText` is used then the `$.detail` JSON object written into the file without double quotes (becomes and invalid JSON)

### Reproduction Steps

The below code generates the mock events;

```python
def generate_data(iteration, unique_id):
return dumps({"iteration": iteration,
"unique-id": unique_id})

def post_some_data(iter=0):
response = client.put_events(
Entries=[
{
'Time': datetime(2015, 1, 1),
'Source': 'test.aws.kinesis',
'DetailType': 'string',
'Detail': generate_data("{}.{}".format(iter, i),
str(time.time()*i*100)),

'EventBusName': 'iot-eb',
'TraceHeader': 'string'
} for i in range(10)
]
)
```

### Versions tried

1. `RuleTargetInput.fromMultilineText`
```typescript
const targetInputTransformation = RuleTargetInput.fromMultilineText(
`{ \"event\": \"${EventField.fromPath('$.detail')}\"}
`)
```

CloudFormation Template:
```yaml
InputTransformer:
InputPathsMap:
detail: $.detail
InputTemplate: |-
"{ \"event\": \"\"}"
""
```

which outputs

```
"""{ \"event\": \"{iteration:1.1,unique-id:162585054621.5018}\"}"
"""{ \"event\": \"{iteration:3.9,unique-id:1463265492438.6333}\"}"
```

2. `RuleTargetInput.fromMultilineText` without quotes
```typescript
const targetInputTransformation = RuleTargetInput.fromMultilineText(
`{event: ${EventField.fromPath('$.detail')}}
`)
```

```yaml
InputTransformer:
InputPathsMap:
detail: $.detail
InputTemplate: |-
"{event: }"
""
```

```
"""{event: {iteration:9998.9,unique-id:1463268536850.662}}"
"""{event: {iteration:9999.1,unique-id:162585392999.7386}}"
```

3. `RuleTargetInput.fromMultilineText` with quotes

```typescript
const targetInputTransformation = RuleTargetInput.fromMultilineText(
'{"event": '+ EventField.fromPath('$.detail')+ ' }\n')
```
generates

```yaml
InputTransformer:
InputPathsMap:
detail: $.detail
InputTemplate: |-
"{\"event\": }"
""
```
which yields the similar output as the other above examples (double quotes removed from the original message

4. `RuleTargetInput.fromObject`

When the following configuration is used, the data is properly written (double quotes are not escaped), however then the new lines cannot be applied.
```
const targetInputTransformation = RuleTargetInput.fromObject({
event: `${EventField.fromPath('$.detail')}`
})
```

```yaml
InputTransformer:
InputPathsMap:
detail: $.detail
InputTemplate: '{"event":}'

```
which appends the records next to each other (as expected)
```text
{"event":{"iteration":"8622.2","unique-id":"325169765164.7032"}}{"event":{"iteration":"8622.0","unique-id":"0.0"}}{"event":{"iteration":"8620.0","unique-id":"0.0"}}{"event":{"iteration":"8622.6","unique-id":"975509295494.1444"}}
```

### Environment

- **CDK CLI Version : 1.110.0**
- **Framework Version:**
- **Node.js Version: 16.3.0**
- **OS : MAC**
- **Language (Version):~4.2.4**

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with RuleTargetInput.fromMultilineText and RuleTargetInput.fromObject, then compare the synthesized CloudFormation InputTransformer shown in the report. Trace how InputTemplate handles EventField.fromPath('$.detail'); done means newline-delimited output remains valid JSON and preserves the quoted fields in the event detail.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, typescript
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.