email.parser.BytesParser.parse() cannot handle binary data that include \x0d \x0a correctly.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
Bug report
Bug description:
I would like to extract a binary file in a multipart MIME file using email.parser.BytesParser, but a byte sequence "0x0d 0x0a" (CR + LF) in the binary file is replaced by "0x0a" (LF). Below is a minimal reproducible example.
from email.parser import BytesParser
from email.policy import default
from io import BytesIO
mime_file_byte_array = b'MIME-Version: 1.0\r\nContent-Type: multipart/mixed; boundary="MIME\
_boundary-1";\r\n\r\n--MIME_boundary-1\r\nContent-Type: application/octet-stream\r\nContent\
-Location: test.bin\r\n\r\na\r\nb\r\n--MIME_boundary-1--\r\n\r\n'
fp = BytesIO(mime_file_byte_array)
parser = BytesParser(policy=default)
msg = parser.parse(fp)
parts = [part for part in msg.walk()]
binary_data = parts[1].get_payload(decode=True)
print('===== Beginning of Original MIME File =====')
print(mime_file_byte_array.decode())
print('===== End of Original MIME File =====')
print('')
print('===== test.bin after parse =====')
print(binary_data)
print('===== test.bin after parse =====')
As can be seen in the fifth line, the multipart MIME file includes a binary file "test.bin". The contents of the binary file is b"a\r\nb".
Therefore, the variable binary_data is supposed to contain b"a\r\nb", but it was actually b"a\nb".
It is probably because TextIOWrapper in BytesParser.parse() translates CR+LF to LF on Linux.
https://github.com/python/cpython/blob/767c89ba7c5a70626df6e75eb56b546bf911b997/Lib/email/parser.py#L103
When I replaced the above line with the line below, this problem was fixed. However, this fix may have a side effect which I cannot foresee.
fp = TextIOWrapper(fp, encoding='ascii', errors='surrogateescape', newline='')
CPython versions tested on:
3.10
Operating systems tested on:
Linux
Linked PRs
- gh-157726
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in Lib/email/parser.py at the TextIOWrapper line referenced by the report, then reproduce the issue with the provided BytesIO example. Compare the parsed payload with the original b"a\r\nb" data and review linked PR gh-157726; the work is done when BytesParser preserves the binary CRLF sequence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100