We are having multiple Task failures for Cloud run, the error message in Cloud Run says "md5 checksum error"
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 398
- Forks
- 181
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
We are getting the md5 checksum error again across multiple pipelines. this error is causing failures for our data pipelines.
Earlier, we received this error and Google provided the following findings:
"GCS doesn't store MD5 checksums for composite objects. A composite object isn't a single entity; it's a collection of individual objects. Verifying the integrity of the composite object itself isn't possible using MD5. This behavior was already in place and there haven’t been any recent changes on the GCS side.
There are chances that earlier object uploaded were not the composite objects and that's why the MD5 checksum consistency check was not the issue for them. From the error, it is confirmed that MD5 checksum verification is done for the composite objects and for composite objects GCS doesn't add any MD5 checksum values.
Since, updating the gcsfs helped here, I believe it might be related to gcsfs releases, please note that gcsfs is not the GCP tool it's a third-party tool. I believe GCS service doesn't have any role in the error you observed."
Solution: We had upgraded the libraries to the latest version which eventually fixed the issue for certain pipelines, however, this time this solution alone is not fixing the issue.
We wanted to connect with you to understand why this error is getting triggered again, is there any change being done on the environment or any release again?
Please let us know at the earliest.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository file, test, or entry point is identified. Start by reviewing the prior Google findings and comparing the affected pipelines' library and environment versions, then reproduce the Cloud Run failure; done means identifying why composite-object MD5 verification is triggered and validating a fix across the failing pipelines.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100