do_insertions produces additional (wrong) token with empty value

(Original issue 1064 created by plotzi76 on 2014-11-27T12:28:14.142634+00:00)

This bug can be seen for all Delegating Lexers:

To reproduce try e.g.

#!python
from pygments.lexers import get_lexer_by_name
lex = get_lexer_by_name("html+php")
list(lex.get_tokens("a<?php>"))   

this yields the following output

#!python
[(Token.Text, u'a'), (Token.Comment.Preproc, u'<?php'), (Token.Operator, u'>'), (Token.Text, u'\n'), (Token.Text, u'')]  

Please note the last token (Token.Text, u'')] which should not be there.

As a workaround I added a check to handle zero length values in the token stream, see below.

#!python
def do_insertions(insertions, tokens):
    """
    Helper for lexers which must combine the results of several
    sublexers.

    ``insertions`` is a list of ``(index, itokens)`` pairs.
    Each ``itokens`` iterable should be inserted at position
    ``index`` into the token stream given by the ``tokens``
    argument.

    The result is a combined token stream.

    TODO: clean up the code here.
    """
    insertions = iter(insertions)
    try:
        index, itokens = next(insertions)
    except StopIteration:
        # no insertions
        for item in tokens:
            yield item
        return

    realpos = None
    insleft = True

    # iterate over the token stream where we want to insert
    # the tokens from the insertion list.
    for i, t, v in tokens:
        # first iteration. store the postition of first item
        if realpos is None:
            realpos = i
        oldi = 0
        while insleft and i + len(v) > index:
            tmpval = v[oldi:index - i]
            # CHANGED! Added check for zero length values
            if len(tmpval) > 0:
                yield realpos, t, tmpval
                realpos += len(tmpval)
            for it_index, it_token, it_value in itokens:
                yield realpos, it_token, it_value
                realpos += len(it_value)
            oldi = index - i
            try:
                index, itokens = next(insertions)
            except StopIteration:
                insleft = False
                break  # not strictly necessary
        yield realpos, t, v[oldi:]
        realpos += len(v) - oldi

    # leftover tokens
    while insleft:
        # no normal tokens, set realpos to zero
        realpos = realpos or 0
        for p, t, v in itokens:
            yield realpos, t, v
            realpos += len(v)
        try:
            index, itokens = next(insertions)
        except StopIteration:
            insleft = False
            break  # not strictly necessary

To be honest, I do not understand the logic for the merging of the two token streams.
I would think that rewritting the DelegatedLexer.get_tokens_unprocessed method
would be the best way.

My proposal for this would be

#!python
    def get_tokens_unprocessed(self, text):
        buf = []
        lastpos = 0       
        for i, t, v in self.language_lexer.get_tokens_unprocessed(text):
            if t is self.needle:
                if not buf:
                    lastpos = i
                buf.append(v)
            else:
                if buf:
                    for ind, tok, val in self.root_lexer.get_tokens_unprocessed(u''.join(buf)):
                        yield lastpos + ind, tok, val
                    del buf[:]
                yield i, t, v

        # buffer might still contain data. 
        if buf:
            for ind, tok, val in self.root_lexer.get_tokens_unprocessed(u''.join(buf)):
                yield lastpos + ind, tok, val

For this solution we do not need the do_insertions at all.

Start with do_insertions and DelegatingLexer.get_tokens_unprocessed, then run the provided html+php reproduction using get_lexer_by_name. Trace how the insertion and delegated token streams are merged; done means the output no longer includes the trailing empty Token.Text value without introducing incorrect positions or tokens.