elastic / elastic/beats

[Meta][GCS] - Improvements and addition of new features to the GCS input

Open
#41,107 9 comments 0 reactions 1 assignee Claimed by @ShourieG View on GitHub
enhancement Epic Filebeat input:GCS needs:triage Team:Security-Service Integrations
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

**Describe the enhancement:** This issue is to track various improvements and addition of new features to the GCS input.

**Describe a specific use case for the enhancement or feature:** With our GCS input being slowly adopted across various integrations, we need to make it more robust and optimal in terms of performance, failure tracking and scalability, hence this Meta issue is created to bring in the necessary changes overtime.

**Improvements:**

- [x] Add metrics to the GCS input. Separate issue [here](https://github.com/elastic/beats/issues/36640)
- [x] [Segregate batch_size from worker count](https://github.com/elastic/beats/issues/44311) (currently worker count is used as the batch_size to distribute jobs evenly)
- [ ] Segregate cursor save op from event publish and add support for detecting elasticsearch acknowledgement signal and use that to update cursor state.
- [x] Improve documentation and explain the impact polling op has on scalability.
- [x] Remove **bucket_timeout** and pass parent program context to bucket operations.

**New Features**
- [x] Add support for SDK level retry mechanism and make it user configurable.
- [ ] Add support for filtering by prefix and glob expressions.
- [ ] Add support for GCS PubSub enabling horizontal scalability of the input.
- [x] Add support for more content-types to the GCS input via content decoders :-
- [x] JSON/NDJSON
- [x] CSV
- [ ] PARQUET
- [ ] TEXT
- [ ] Add support for state tracking via optional **startOffset** (user configurable with certain ordering limitations)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.