HTML Entities cause split after the next character

I noticed this behaviour in https://github.com/rusterlium/html5ever_elixir, but it seems to be part of this excellent project.

What I see is that HTML Entities cause a break split after the next regular character. So ""ABC">DE" will return the tokens "\"', "A", "BC", "\"", ">", "D", "E".

I expected one character token "\"ABC\">DE" instead.

I wrote a program (with ChatGPT as I am not very familiar with rust) to demonstrate. If there is an error in there, please let me know.

use tendril::SliceExt;
use html5ever::tokenizer::{BufferQueue, TokenSinkResult, TokenSink, Token};

struct TokenPrinter;

impl TokenSink for TokenPrinter {
    type Handle = (); // This can be unit type if you're not using handles in your implementation

    fn process_token(&mut self, token: Token, _line_number: u64) -> TokenSinkResult<Self::Handle> {
        match token {
            // Match and print tokens here
            _ => println!("{:?}", token),
        }
        TokenSinkResult::Continue
    }
}

fn main() {
    // The HTML string you want to tokenize
    let html_string = r#"&quot;ABC&quot;&gt;DE"#;

    // Tokenize the HTML string and print the tokens
    let mut tokenizer = html5ever::tokenizer::Tokenizer::new(TokenPrinter, Default::default());
    let mut input = BufferQueue::new();
    input.push_back(html_string.to_tendril());
    let _ = tokenizer.feed(&mut input);
    tokenizer.end();
}

gives the output

CharacterTokens(Tendril<UTF8>(inline: "\""))
CharacterTokens(Tendril<UTF8>(inline: "A"))
CharacterTokens(Tendril<UTF8>(inline: "BC"))
CharacterTokens(Tendril<UTF8>(inline: "\""))
CharacterTokens(Tendril<UTF8>(inline: ">"))
CharacterTokens(Tendril<UTF8>(inline: "D"))
CharacterTokens(Tendril<UTF8>(inline: "E"))
EOFToken

If you need any more information, please let me know.

Reproduce the example using html5ever::tokenizer::Tokenizer, BufferQueue, and the TokenSink implementation shown in the report. Trace how tokenizer.feed and tokenizer.end emit CharacterTokens for entity-containing input; done means the sample produces one character token containing the expected combined string instead of splitting after entities.