redpanda-data / redpanda-data/connect

Kafka-> S3- invalid header field value for \"X-Amz-Meta-Kafka_key\"

Open
#1,666 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

aws bughancement outputs ux
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Hello!
I'm afraid the thing I'm reporting is not a bug per se, but I find it exceptionally difficult to figure out the reason for the error.
I'm using pretty standard configuration to consume messages from Kafka (attached below), but I'm getting rather cryptic error

Failed to send message to aws_s3: RequestError: send request failed
caused by: Put \"<HTTP S3 URL>\": net/http: invalid header field value for \"X-Amz-Meta-Kafka_key\"

I'm using jeffail/benthos:4.9.1.

Can it be caused by unprocessable keys for key/value pairs in Kafka?

    input:
      kafka:
        addresses:
          - ${KAFKA_ADDRESSES}
        client_id: ${CLIENT_ID}
        commit_period: 3s
        consumer_group: ${CONSUMER_GROUP}
        group:
          heartbeat_interval: 3s
          rebalance_timeout: 60s
          session_timeout: 10s
        checkpoint_limit: 1
        batching:
          count: 1024
          period: 1s
        max_processing_period: 2s
        start_from_oldest: true
        target_version: 3.2.0
        topics:
          - ${INPUT_TOPIC}

    pipeline:
      processors:
        - group_by_value:
            value: ${! meta("kafka_partition") }
        - archive:
            format: binary

    output:
      aws_s3:
        bucket: ${S3_BUCKET}
        content_encoding: ""
        content_type: application/octet-stream
        credentials:
          id: ${AWS_ACCESS_KEY_ID}
          profile: ""
          role: ""
          role_external_id: ""
          secret: ${AWS_SECRET_ACCESS_KEY}
          token: ""
        endpoint: ""
        force_path_style_urls: false
        path: ${S3_PATH}
        region: ${AWS_REGION}
        timeout: 5s

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported failure with the provided Kafka input, pipeline, and aws_s3 output configuration on the referenced Benthos version. Trace the handling of the Kafka key through the S3 request and identify what produces the invalid X-Amz-Meta-Kafka_key header; done means the cause is established and the failure has a verified resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.