[Meta][GCS] - Improvements and addition of new features to the GCS input
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 370
Description
**Describe the enhancement:** This issue is to track various improvements and addition of new features to the GCS input.
**Describe a specific use case for the enhancement or feature:** With our GCS input being slowly adopted across various integrations, we need to make it more robust and optimal in terms of performance, failure tracking and scalability, hence this Meta issue is created to bring in the necessary changes overtime.
**Improvements:**
- [x] Add metrics to the GCS input. Separate issue [here](https://github.com/elastic/beats/issues/36640)
- [x] [Segregate batch_size from worker count](https://github.com/elastic/beats/issues/44311) (currently worker count is used as the batch_size to distribute jobs evenly)
- [ ] Segregate cursor save op from event publish and add support for detecting elasticsearch acknowledgement signal and use that to update cursor state.
- [x] Improve documentation and explain the impact polling op has on scalability.
- [x] Remove **bucket_timeout** and pass parent program context to bucket operations.
**New Features**
- [x] Add support for SDK level retry mechanism and make it user configurable.
- [ ] Add support for filtering by prefix and glob expressions.
- [ ] Add support for GCS PubSub enabling horizontal scalability of the input.
- [x] Add support for more content-types to the GCS input via content decoders :-
- [x] JSON/NDJSON
- [x] CSV
- [ ] PARQUET
- [ ] TEXT
- [ ] Add support for state tracking via optional **startOffset** (user configurable with certain ordering limitations)
Contributor guide
Assessment
This issue has not been assessed yet.