apache / apache/beam

GCSFileSystem is not facilitating injection of the gcs url

Open
#21,255 0 comments 0 reactions 0 assignees View on GitHub
gcp improvement io P2 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

Inside `apache_beam.io.gcp.gcsfilesystem.GCSFileSystem` is directly instantiating for every operations an instance of `apache_beam.io.gcp.gcsio.GcsIO` without allowing anyone to make use of the flexibility of instantiating with a custom client storage.

As you can see inside `GcsIO` there's an optional parameter called `storage_client` and that would give us the possibility to overwrite the storage client by example depending of the run level and potentially make use of a Gcs emulator locally when testing the infrastructure.

Right now the only way we have to overwrite the gcs url is by monkey patching the default storage which is not most clean way of doing it.

 

Potential solution:

If `GCSFileSystem` was transformed by example by adding a single class method to generate every one of thse `GcsIO`, that would give us the possibility to create a custom `GCSFileSystem` class, inheriting from it, without duplicating all of it's code and to inject our desired url depending of the run level.

Imported from Jira [BEAM-13250](https://issues.apache.org/jira/browse/BEAM-13250). Original Jira may contain additional context.
Reported by: mlhamel.

Contributor guide

Open the contributing guide

Research direction

Start with apache_beam.io.gcp.gcsfilesystem.GCSFileSystem and inspect where its operations instantiate apache_beam.io.gcp.gcsio.GcsIO. Read the GcsIO storage_client parameter and determine how a subclass or factory can provide a custom client without duplicating GCSFileSystem; done means callers can use an alternate GCS URL or emulator without monkey-patching.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.