refactor the lexer

At the moment the lexer uses one huge function (`tokenize`) to tokenize the code. The code could become cleaner, if this could be refactored into several functions. For this some kind of `LexerState`-struct would be necessary.

Start by locating the lexer entry point named `tokenize` and inspect its current responsibilities and tokenization behavior. Identify suitable function boundaries and the required `LexerState` structure; the refactor is done when the lexer is split into several functions without changing its behavior.

Written by the indexing model from the issue text.