PipedreamHQ / PipedreamHQ/pipedream
Configurable flush timers for `send` events.
- Dominant language
- JavaScript
- Stars
- 11.7k
- Forks
- 5.8k
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 102
Description
**Feature request**
For low-volume events, being able to configure longer flush timers for the `$send` actions would provide a way to (further) reduce the write calls to things like an s3 bucket. This way the bucket owners save in PUT operations to the buckets and Pipedreams saves on API calls to AWS.
**Describe the solution you'd like**
Having a configurable timer that allows me to buffer x number of minutes before writing to the s3 buckets, being able to collect more events in a single write. This would probably improve efficiency on the AWS calls. Internally you might want to set up a max number of buffered bytes and/or max number of events that would trigger a write if the flush timer didn't expire. Documenting those limits would be good but I don't think they need to be user configurable.
**Do you have a workaround?**
No work-around but not a deal-breaker only. This would be an improvement for low-volume writes.
**Comparable features in other tools?**
New to pipedream so nothing to compare it to at the moment.
**Additional context**
Im testing pipedream as a webhook resource that lets me collect info from users of a wifi captive portal. I don't need minute-by-minute availability of the logged data, just want to store it for longer-term analytics. Saving API calls benefits everyone I believe.
Contributor guide
Assessment
This issue has not been assessed yet.