React on input from GCS for newer files

At the moment the GCS supports reading from the bucket for all files that are present. However if i have a bucket which is partitioned by the date for example, it would read all the files from all the buckets at every execution

e.g.:

gs://bucket_name/prefix_name/2021-10-20/*.csv
gs://bucket_name/prefix_name/2021-10-19/*.csv
gs://bucket_name/prefix_name/2021-10-18/*.csv

There is the option to delete the objects (i.e. delete_objects after benthos has read them. However this approach is not suitable for all usecase as it is deleting data.

It would be good to have a way to make benthos be aware of the state so it doesn't read the same files again unless specified otherwise.

Or any other idea like for example use a benthos wildcard for the folder names which can be calculated with bloblang.

The issue points to the GCS input documentation at components/inputs/gcp_cloud_storage; start by reviewing how object discovery and delete_objects currently work. Define how processed-object state or date-based selection should behave without deleting data, and verify that previously read files are not read again unless explicitly requested.