redpanda-data / redpanda-data/benthos
http_client inside read_until input does not stop on error even with retries=0
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 571
- Forks
- 120
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 18
Description
To reproduce, use the following snippets for client and server
client
input:
read_until:
input:
http_client:
url: "http://localhost:8088/retry-test"
verb: GET
timeout: 10s
retries: 0
stream:
enabled: false
reconnect: false
scanner:
to_the_end: {}
check: true
restart_input: false
pipeline:
processors:
- noop: {}
output:
stdout: {}
server
http:
enabled: true
address: ":localhost:8089"
logger:
level: INFO
format: json
add_timestamp: true
input:
http_server:
address: "localhost:8088"
path: /retry-test
allowed_verbs:
- GET
timeout: 60s
rate_limit: ""
pipeline:
processors:
- log:
level: INFO
message: "Received message"
- sleep:
duration: 15s
- mapping: |
root = {"content": "test"}
output:
sync_response:
status: "200"
headers:
Content-Type: application/json
This is supposed to simulate a download from an HTTP server that runs into a timeout. The file is supposed to be download once and then processed by a pipeline. My expectation is that this stops after the first attempt, either by retries=0 on the http_client or the true condition on the read_until. However, this just keep sending requests to the server:
Server output
{"@service":"redpanda-connect","benthos_version":"v4.47.1","level":"info","msg":"Running main config from specified file","path":"/benthos/mock_server.yaml","time":"2025-02-21T12:57:07Z"}
{"@service":"redpanda-connect","level":"info","msg":"Listening for HTTP requests at: http://localhost:8089","time":"2025-02-21T12:57:07Z"}
{"@service":"redpanda-connect","label":"","level":"info","msg":"Receiving HTTP messages at: http://localhost:8088/retry-test","path":"root.input","time":"2025-02-21T12:57:07Z"}
{"@service":"redpanda-connect","level":"info","msg":"Launching a Redpanda Connect instance, use CTRL+C to close","time":"2025-02-21T12:57:07Z"}
{"@service":"redpanda-connect","label":"","level":"info","msg":"Output type sync_response is now active","path":"root.output","time":"2025-02-21T12:57:07Z"}
{"@service":"redpanda-connect","custom_source":true,"label":"","level":"info","msg":"Received message","path":"root.pipeline.processors.0","time":"2025-02-21T12:57:27Z"}
{"@service":"redpanda-connect","custom_source":true,"label":"","level":"info","msg":"Received message","path":"root.pipeline.processors.0","time":"2025-02-21T12:57:38Z"}
{"@service":"redpanda-connect","custom_source":true,"label":"","level":"info","msg":"Received message","path":"root.pipeline.processors.0","time":"2025-02-21T12:57:48Z"}
As far as I understand the documentation, it may be valid that the read_until does not cause a stop even with the condition always evaluating to true, as it checks the condition after producing a message. However, the http_client should not retry as this is being set to 0 retries
The only thing that limits the permanent retry is setting the idle_timeout on the read_until. However, in my opinion this should not apply at all as the input is not idle. I suspect this is only checking whether new messages are produced?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the client and server configuration snippets to reproduce the repeated requests and timeout. Trace the read_until and http_client input handling to determine how retries=0 and the true check interact. Done means the first timed-out request does not lead to further requests, while the intended read_until behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100