redpanda-data / redpanda-data/benthos
[request] add schedule processor
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 571
- Forks
- 120
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 18
Description
Hi,
It would be nice to add schedule processor to consume the child input with defined interval or cron.
Use case: read paginated http response with interval.
I can do it only with external scheduler (like cron) now. Or maybe is it possible with current capabilities?
Example:
input:
# need repeat this block every hour
read_until:
check: '@offset >= 5'
input:
generate:
count: 0
interval: 1s
mapping: 'root = null'
processors:
- mapping: |
meta n = 5
meta offset = counter(min: 0)
- http:
url: https://httpbin.org/links/${! metadata("n") }/${! metadata("offset") }
verb: GET
pipeline:
processors:
- xml:
operator: to_json
- mapping: |
root = json("html.body.a")
- unarchive:
format: json_array
- mutation: |
root.page = @offset
output:
stdout: {}
Possible solutions:
- add
inputfield to thegenerateinput - add
scheduleinput
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing generate input and read_until behavior described in the example, then compare the two proposed scheduling approaches: nesting an input under generate or adding a schedule input. Done means a tested way to repeat child input on an interval or cron without an external scheduler, including the paginated HTTP use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100