fsspec / fsspec/gcsfs

We are having multiple Task failures for Cloud run, the error message in Cloud Run says "md5 checksum error"

Open
#661 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

We are getting the md5 checksum error again across multiple pipelines. this error is causing failures for our data pipelines.
Earlier, we received this error and Google provided the following findings:
"GCS doesn't store MD5 checksums for composite objects. A composite object isn't a single entity; it's a collection of individual objects. Verifying the integrity of the composite object itself isn't possible using MD5. This behavior was already in place and there haven’t been any recent changes on the GCS side.
There are chances that earlier object uploaded were not the composite objects and that's why the MD5 checksum consistency check was not the issue for them. From the error, it is confirmed that MD5 checksum verification is done for the composite objects and for composite objects GCS doesn't add any MD5 checksum values.
Since, updating the gcsfs helped here, I believe it might be related to gcsfs releases, please note that gcsfs is not the GCP tool it's a third-party tool. I believe GCS service doesn't have any role in the error you observed."
Solution: We had upgraded the libraries to the latest version which eventually fixed the issue for certain pipelines, however, this time this solution alone is not fixing the issue.
We wanted to connect with you to understand why this error is getting triggered again, is there any change being done on the environment or any release again?
Please let us know at the earliest.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository file, test, or entry point is identified. Start by reviewing the prior Google findings and comparing the affected pipelines' library and environment versions, then reproduce the Cloud Run failure; done means identifying why composite-object MD5 verification is triggered and validating a fix across the failing pipelines.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.