redpanda-data / redpanda-data/connect
Kafka-> S3- invalid header field value for \"X-Amz-Meta-Kafka_key\"
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
Hello!
I'm afraid the thing I'm reporting is not a bug per se, but I find it exceptionally difficult to figure out the reason for the error.
I'm using pretty standard configuration to consume messages from Kafka (attached below), but I'm getting rather cryptic error
Failed to send message to aws_s3: RequestError: send request failed
caused by: Put \"<HTTP S3 URL>\": net/http: invalid header field value for \"X-Amz-Meta-Kafka_key\"
I'm using jeffail/benthos:4.9.1.
Can it be caused by unprocessable keys for key/value pairs in Kafka?
input:
kafka:
addresses:
- ${KAFKA_ADDRESSES}
client_id: ${CLIENT_ID}
commit_period: 3s
consumer_group: ${CONSUMER_GROUP}
group:
heartbeat_interval: 3s
rebalance_timeout: 60s
session_timeout: 10s
checkpoint_limit: 1
batching:
count: 1024
period: 1s
max_processing_period: 2s
start_from_oldest: true
target_version: 3.2.0
topics:
- ${INPUT_TOPIC}
pipeline:
processors:
- group_by_value:
value: ${! meta("kafka_partition") }
- archive:
format: binary
output:
aws_s3:
bucket: ${S3_BUCKET}
content_encoding: ""
content_type: application/octet-stream
credentials:
id: ${AWS_ACCESS_KEY_ID}
profile: ""
role: ""
role_external_id: ""
secret: ${AWS_SECRET_ACCESS_KEY}
token: ""
endpoint: ""
force_path_style_urls: false
path: ${S3_PATH}
region: ${AWS_REGION}
timeout: 5s
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported failure with the provided Kafka input, pipeline, and aws_s3 output configuration on the referenced Benthos version. Trace the handling of the Kafka key through the S3 request and identify what produces the invalid X-Amz-Meta-Kafka_key header; done means the cause is established and the failure has a verified resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100