Support copying GCs objects using rewrite operation
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently Java GCS FileSystem users GCS copy() operation to copy objects.
https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java#L744
According to GCS website this can result in HTTP 413 errors if copying takes more than 30 seconds: https://cloud.google.com/storage/docs/json_api/v1/status-codes
The solution suggested there is to use the GCS rewrite() method.
We should add support for this.
Imported from Jira [BEAM-3413](https://issues.apache.org/jira/browse/BEAM-3413). Original Jira may contain additional context.
Reported by: chamikara.
Contributor guide
Research direction
Start in sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java around the copy() operation at the referenced line, and compare the GCS copy and rewrite APIs. Done means GCS object copying uses rewrite support for operations that can exceed the copy request limit, with the existing behavior covered by the relevant GcsUtil tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100