redpanda-data / redpanda-data/connect

Different behavior in JSON environment variable parsing during testing vs. linting

Open
#1,965 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs investigation
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Given the files:

.env

MY_JSON="[{ \"id\": 1234 }]"

benthos_benthos_test.yml

tests:
  - name: dummy
    target_processors: "/input/processors"
    environment:
      MY_JSON: '[{ \"id\": 1234 }]'
    input_batch:
      - content: ""
    output_batches:
      - - json_equals: [{ "id": 1234 }]

benthos.yml

input:
  generate:
    count: 1
    interval: ""
    mapping: |
      root = ""
  processors:
    - mapping: |
        root = "${MY_JSON}".parse_json()

Scenario 1

When I run test

$ benthos test ./...
Test 'benthos.yml' succeeded

But when I run lint

$ benthos -e .env lint ./... && echo succeeded
benthos.yml(9,29) line 1 char 13: expected line break
  |
1 | root = "[{ "id": 1234 }]".parse_json()
  |             ^---

Scenario 2

Changing benthos.yml to

input:
  generate:
    count: 1
    interval: ""
    mapping: |
      root = ""
  processors:
    - mapping: |
        root = ${MY_JSON}

When I run test:

$ benthos test ./...
Failed to execute test target 'benthos.yml': test case 0 failed: failed to initialise processors '/input/processors': failed to initialise processor index '0': failed to init processor <no label>: failed to parse bloblang mapping '': expected query, got: \"id\

But when I run lint:

$ benthos -e .env lint ./... && echo succeeded
succeeded

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the two scenarios using .env, benthos_benthos_test.yml, and benthos.yml, running benthos test ./... and benthos -e .env lint ./.... Compare how environment variables are expanded and parsed in testing versus linting; done should mean the discrepancy is resolved or its intended behavior is explicitly established.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.