Specify GCS Object Version in apache_beam.io.gcp.gcsio
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
I would like to specify a generation when accessing a gcs object via the beam filesystem.
Via the cli with the gsutil command a specific version can be access by the following syntax.
```
gsutil cp gs://{bucket}/{object_path}#{generation} .
```
So the corresponding python code would look something like this
```
with apache_beam.io.filesystems.open("gs://{bucket}/{object_path}#{generation}") as f:
pass
```
Fortunately, the [StorageObjectsGetRequest](https://github.com/apache/beam/blob/14862ccbdf2879574b6ce49149bdd7c9bf197322/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_messages.py#L2133) can already be passed a generation.
However, this is *****not done***** within the [GcsDownloader](https://github.com/apache/beam/blob/14862ccbdf2879574b6ce49149bdd7c9bf197322/sdks/python/apache_beam/io/gcp/gcsio.py#L611).
I think when [parsing the GCS path](https://github.com/apache/beam/blob/14862ccbdf2879574b6ce49149bdd7c9bf197322/sdks/python/apache_beam/io/gcp/gcsio.py#L583) the generation should be extracted as well.
Imported from Jira [BEAM-14165](https://issues.apache.org/jira/browse/BEAM-14165). Original Jira may contain additional context.
Reported by: l_karls.
Contributor guide
Assessment
This issue has not been assessed yet.