[Bug]: AttributeError: 'HttpError' object has no attribute 'message' GcsIO
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
### What happened?
Beam version: 2.45.0
While performing write operations using the DoFn below, the GcsIO connector returns the following traceback:
```
Error message from worker: generic::unknown: Traceback (most recent call last):
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 624, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/project/pipelines/utilities/writers.py", line 68, in process
with self.gcs.open(file_name, "wb") as f:
File "/usr/local/lib/python3.10/site-packages/apache_beam/io/filesystemio.py", line 215, in close
self._uploader.finish()
File "/usr/local/lib/python3.10/site-packages/apache_beam/io/gcp/gcsio.py", line 829, in finish
self._upload_thread.last_error.message) # pylint: disable=raising-bad-type
AttributeError: 'HttpError' object has no attribute 'message'
```
The problem appears to be transient and stopped when HttpErrors stopped occurring.
```
class WriteFilesToGCS(DoFn):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def start_bundle(self):
self.gcs = gcsio.GcsIO()
def finish_bundle(self):
del self.gcs
def process(self, element: Tuple[str, str], *args, **kwargs):
file_name, file_content
# Write the file to GCS
with self.gcs.open(file_name, "wb") as f:
with GzipFile(fileobj=f, mode="wb") as gz:
gz.write(file_content.encode("utf-8"))
```
### Issue Priority
Priority: 3 (minor)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner
Contributor guide
Research direction
Start in apache_beam/io/gcp/gcsio.py at the GCS upload finish path shown in the traceback, and inspect how an HttpError from the upload thread is reported. Reproduce or test the failing error path; done means the underlying upload error is surfaced without a secondary AttributeError about the missing message attribute.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100