tensorflow / tensorflow/tensorboard
GFile S3 reader may corrupt files containing multibyte characters or CR/LF
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
In our stub GFile implementation, the S3 reader obtains chunks of n bytes and then decodes them as strings. This risks corrupting data or at least throwing decoding errors, because the byte chunk may end in the middle of a multibyte character (and correspondingly, the next chunk would begin in the middle of that character).
Thus the chunks cannot be decoded independently. Any partial character must be detected and removed from the end of each chunk, and prepended to the subsequent chunk before decoding.
CRLF (\r\n) is effectively a multibyte character in this context, because Python always "decodes" it to just \n.
(Followup to #2791 and #2777)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read tensorboard/compat/tensorflow_stub/io/gfile.py around line 268, focusing on how the S3 reader chunks and decodes data. Check the related context in issues #2791 and #2777. Done means multibyte characters and CR/LF sequences remain intact across chunk boundaries without decoding errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100