apache / apache/beam

Allow upload gzipped files via apache_beam.io.gcp.gcsio.GcsIO with proper content-encoding

Open
#19,413 0 comments 0 reactions 0 assignees View on GitHub
gcp improvement io P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

To reduce the size of uploaded files we decided to gzip it before upload. Unfortunately, we noticed that we don't have content-encoding 'gzip' in the uploaded files metadata. I rechecked the code and noticed that there is no way to pass gzip encoding on
```

apache_beam.io.gcp.gcsio.GcsIO.open()
```

Also, I noticed that apache_beam.io.gcp.gcsio.GcsUploader doesn't support uploading for gzipped files.

To resolve this problem we need to allow pass gzip_encoded option, which can be passed to apitools.base.py.transfer on
```

GcsUploader.__init__()

```

Is there any possibility that you apply the required changes soon?

*What steps to reproduce the problem?*
1. Prepare gzip encoded file for example pdf
2. Upload it to GCS using
```

from apache_beam.io.gcp.gcsio import GcsIO

def upload_gzipped_pdf(gzipped_pdf, path)
with GcsIO().open(path,
'w') as f:
f.write(gzipped_pdf)

```

3. Try to download uploaded file via browser

*What is the expected result?*
I see the file content properly

*What happens instead?*
I have a broken document

 

*Possible resolution after implementing expected changes*
```

from apache_beam.io.gcp.gcsio import GcsIO

def upload_gzipped_pdf(gzipped_pdf, path)
with GcsIO().open(path,
'w', gzip_encoded=True) as f:
f.write(gzipped_pdf)

```

Imported from Jira [BEAM-7411](https://issues.apache.org/jira/browse/BEAM-7411). Original Jira may contain additional context.
Reported by: p35.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.