MIME lexer has difficulties when used by HTTP lexer for multipart/mixed payloads
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 885
- PR merge metrics
- No merged PRs in 30d
Description
Now that we've added a MIME lexer, it's getting used not only by email lexing, but also by the HTTP lexer, and I've uncovered an instance where it's not working. It's most easily demonstrated by two additional test example files (one passes, one fails).
A simple multipart/form-data PUT succeeds, but a multipart/mixed POST does not succeed (see below). I would expect both of these examples to be successfully lexed/formatted. (The lexer, in the second case, seems to generate an error token immediately upon hitting the first - character for the first boundary.)
This example passes (a multipart form HTTP trace example):
PUT /put HTTP/1.1
Host: httpbin.org
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip,deflate
Accept: */*
Connection: keep-alive
Content-Length: 180
Content-Type: multipart/form-data; boundary=xxBOUNDARYxx
--xxBOUNDARYxx
Content-Disposition: form-data; name="Sample file"; filename="file.txt"
This is a sample text file
with some text content.
--xxBOUNDARYxx--
However, this "multipart/mixed" example does not pass:
POST /post HTTP/1.1
Host: httpbin.org
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip,deflate
Accept: */*
Connection: keep-alive
Content-Length: 438
Content-Type: multipart/mixed; boundary=xxBOUNDARYxx
--xxBOUNDARYxx
Content-Type: application/json
{"IsHidden": false, "IsLocked": false, "ShortTitle": "Test", "Type": 1, "DueDate": null, "Url": "/content/extensibility/EXT-104/file.txt", "StartDate": null, "TopicType": 1, "EndDate": null, "Title": "Test topic content"}
--xxBOUNDARYxx
Content-Disposition: form-data; name=""; filename="file.txt"
Content-Type: text/plain
This is a sample text file
with some text content.
--xxBOUNDARYxx--
Adding both these as test file examples, and the second one fails (tested under py3.7.5), with this test failure output:
____________________________ test_examplefile[http_multipartmixed_example] _____________________________
filename = 'http_multipartmixed_example'
@pytest.mark.parametrize('filename', get_example_files())
def test_examplefile(filename):
if os.name == 'java' and filename in BAD_FILES_FOR_JYTHON:
pytest.skip('%s is a known bad file on Jython' % filename)
absfn = os.path.join(TESTDIR, 'examplefiles', filename)
with open(absfn, 'rb') as f:
text = f.read()
try:
utext = text.decode('utf-8')
except UnicodeError:
utext = text.decode('latin1')
lx = None
if '_' in filename:
try:
lx = get_lexer_by_name(filename.split('_')[0])
except ClassNotFound:
pass
if lx is None:
try:
lx = get_lexer_for_filename(absfn, code=utext)
except ClassNotFound:
raise AssertionError('file %r has no registered extension, '
'nor is of the form <lexer>_filename '
'for overriding, thus no lexer found.'
% filename)
text = text.replace(b'\r\n', b'\n')
text = text.strip(b'\n') + b'\n'
try:
text = text.decode('utf-8')
if text.startswith(u'\ufeff'):
text = text[len(u'\ufeff'):]
except UnicodeError:
text = text.decode('latin1')
ntext = []
tokens = []
import time
t1 = time.time()
for type, val in lx.get_tokens(text):
ntext.append(val)
> assert type != Error, \
'lexer %s generated error token for %s: %r at position %d' % \
(lx, absfn, val, len(u''.join(ntext)))
E AssertionError: lexer <pygments.lexers.HttpLexer> generated error token for /Users/vhaag/src/pygments/tests/examplefiles/http_multipartmixed_example: '-' at position 213
E assert Token.Error != Token.Error
tests/test_examplefiles.py:116: AssertionError
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the example-file test in tests/test_examplefiles.py against the multipart examples described in the issue, especially the failing http_multipartmixed_example. Trace the HTTP and MIME lexer handling at the first boundary dash, then verify that both multipart/form-data and multipart/mixed examples produce no Error tokens and are successfully lexed/formatted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100