apache / apache/beam

Specify GCS Object Version in apache_beam.io.gcp.gcsio

Open
#21,518 0 comments 0 reactions 0 assignees View on GitHub
gcp improvement io P2 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I would like to specify a generation when accessing a gcs object via the beam filesystem.
Via the cli with the gsutil command a specific version can be access by the following syntax.

```

gsutil cp gs://{bucket}/{object_path}#{generation} .

```

So the corresponding python code would look something like this
```

with apache_beam.io.filesystems.open("gs://{bucket}/{object_path}#{generation}") as f:
pass

```

Fortunately, the [StorageObjectsGetRequest](https://github.com/apache/beam/blob/14862ccbdf2879574b6ce49149bdd7c9bf197322/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_messages.py#L2133) can already be passed a generation.
However, this is *****not done***** within the [GcsDownloader](https://github.com/apache/beam/blob/14862ccbdf2879574b6ce49149bdd7c9bf197322/sdks/python/apache_beam/io/gcp/gcsio.py#L611).

I think when [parsing the GCS path](https://github.com/apache/beam/blob/14862ccbdf2879574b6ce49149bdd7c9bf197322/sdks/python/apache_beam/io/gcp/gcsio.py#L583) the generation should be extracted as well.

Imported from Jira [BEAM-14165](https://issues.apache.org/jira/browse/BEAM-14165). Original Jira may contain additional context.
Reported by: l_karls.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.