apache / apache/beam

Add support for path with braces for Google Cloud Storage

Open Beginner friendly
#18,505 0 comments 0 reactions 0 assignees View on GitHub
gcp improvement io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

At the moment we can not use braces in Google Cloud Storage paths, as explained [here](https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific).

The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function.

`gs://bucket/{file1,file2,file3}` should match `gs://bucket/file1`, `gs://bucket/file2` and `gs://bucket/file3`

Imported from Jira [BEAM-3224](https://issues.apache.org/jira/browse/BEAM-3224). Original Jira may contain additional context.
Reported by: benjzer.

Contributor guide

Open the contributing guide

Research direction

Start in sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java, focusing on the wildcardToRegexp function and how it expands Java glob patterns. Verify the change against the stated gs://bucket/{file1,file2,file3} example; done means each listed path matches separately.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.