apache / apache/beam

[Bug]: AttributeError: 'HttpError' object has no attribute 'message' GcsIO

Open
#25,911 2 comments 2 reactions 0 assignees View on GitHub
awaiting triage bug dataflow io P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

### What happened?

Beam version: 2.45.0

While performing write operations using the DoFn below, the GcsIO connector returns the following traceback:

```
Error message from worker: generic::unknown: Traceback (most recent call last):
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 624, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/project/pipelines/utilities/writers.py", line 68, in process
with self.gcs.open(file_name, "wb") as f:
File "/usr/local/lib/python3.10/site-packages/apache_beam/io/filesystemio.py", line 215, in close
self._uploader.finish()
File "/usr/local/lib/python3.10/site-packages/apache_beam/io/gcp/gcsio.py", line 829, in finish
self._upload_thread.last_error.message) # pylint: disable=raising-bad-type
AttributeError: 'HttpError' object has no attribute 'message'
```

The problem appears to be transient and stopped when HttpErrors stopped occurring.

```
class WriteFilesToGCS(DoFn):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)

def start_bundle(self):
self.gcs = gcsio.GcsIO()

def finish_bundle(self):
del self.gcs

def process(self, element: Tuple[str, str], *args, **kwargs):
file_name, file_content

# Write the file to GCS
with self.gcs.open(file_name, "wb") as f:
with GzipFile(fileobj=f, mode="wb") as gz:
gz.write(file_content.encode("utf-8"))
```

### Issue Priority

Priority: 3 (minor)

### Issue Components

- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Research direction

Start in apache_beam/io/gcp/gcsio.py at the GCS upload finish path shown in the traceback, and inspect how an HttpError from the upload thread is reported. Reproduce or test the failing error path; done means the underlying upload error is surfaced without a secondary AttributeError about the missing message attribute.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.