pygments / pygments/pygments

MIME lexer has difficulties when used by HTTP lexer for multipart/mixed payloads

Open
#1,292 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

T-bug
Dominant language
Python
Stars
2.2k
Forks
885
PR merge metrics
No merged PRs in 30d

Description

Now that we've added a MIME lexer, it's getting used not only by email lexing, but also by the HTTP lexer, and I've uncovered an instance where it's not working. It's most easily demonstrated by two additional test example files (one passes, one fails).

A simple multipart/form-data PUT succeeds, but a multipart/mixed POST does not succeed (see below). I would expect both of these examples to be successfully lexed/formatted. (The lexer, in the second case, seems to generate an error token immediately upon hitting the first - character for the first boundary.)

This example passes (a multipart form HTTP trace example):

PUT /put HTTP/1.1
Host: httpbin.org
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip,deflate
Accept: */*
Connection: keep-alive
Content-Length: 180
Content-Type: multipart/form-data; boundary=xxBOUNDARYxx

--xxBOUNDARYxx
Content-Disposition: form-data; name="Sample file"; filename="file.txt"

This is a sample text file
with some text content.
--xxBOUNDARYxx--

However, this "multipart/mixed" example does not pass:

POST /post HTTP/1.1
Host: httpbin.org
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip,deflate
Accept: */*
Connection: keep-alive
Content-Length: 438
Content-Type: multipart/mixed; boundary=xxBOUNDARYxx

--xxBOUNDARYxx
Content-Type: application/json

{"IsHidden": false, "IsLocked": false, "ShortTitle": "Test", "Type": 1, "DueDate": null, "Url": "/content/extensibility/EXT-104/file.txt", "StartDate": null, "TopicType": 1, "EndDate": null, "Title": "Test topic content"}
--xxBOUNDARYxx
Content-Disposition: form-data; name=""; filename="file.txt"
Content-Type: text/plain

This is a sample text file
with some text content.
--xxBOUNDARYxx--

Adding both these as test file examples, and the second one fails (tested under py3.7.5), with this test failure output:

____________________________ test_examplefile[http_multipartmixed_example] _____________________________

filename = 'http_multipartmixed_example'

    @pytest.mark.parametrize('filename', get_example_files())
    def test_examplefile(filename):
        if os.name == 'java' and filename in BAD_FILES_FOR_JYTHON:
            pytest.skip('%s is a known bad file on Jython' % filename)
    
        absfn = os.path.join(TESTDIR, 'examplefiles', filename)
        with open(absfn, 'rb') as f:
            text = f.read()
        try:
            utext = text.decode('utf-8')
        except UnicodeError:
            utext = text.decode('latin1')
    
        lx = None
        if '_' in filename:
            try:
                lx = get_lexer_by_name(filename.split('_')[0])
            except ClassNotFound:
                pass
        if lx is None:
            try:
                lx = get_lexer_for_filename(absfn, code=utext)
            except ClassNotFound:
                raise AssertionError('file %r has no registered extension, '
                                     'nor is of the form <lexer>_filename '
                                     'for overriding, thus no lexer found.'
                                     % filename)
    
        text = text.replace(b'\r\n', b'\n')
        text = text.strip(b'\n') + b'\n'
        try:
            text = text.decode('utf-8')
            if text.startswith(u'\ufeff'):
                text = text[len(u'\ufeff'):]
        except UnicodeError:
            text = text.decode('latin1')
        ntext = []
        tokens = []
        import time
        t1 = time.time()
        for type, val in lx.get_tokens(text):
            ntext.append(val)
>           assert type != Error, \
                'lexer %s generated error token for %s: %r at position %d' % \
                (lx, absfn, val, len(u''.join(ntext)))
E           AssertionError: lexer <pygments.lexers.HttpLexer> generated error token for /Users/vhaag/src/pygments/tests/examplefiles/http_multipartmixed_example: '-' at position 213
E           assert Token.Error != Token.Error

tests/test_examplefiles.py:116: AssertionError

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the example-file test in tests/test_examplefiles.py against the multipart examples described in the issue, especially the failing http_multipartmixed_example. Trace the HTTP and MIME lexer handling at the first boundary dash, then verify that both multipart/form-data and multipart/mixed examples produce no Error tokens and are successfully lexed/formatted.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.