How to create a custom lexer?

Open
#1,463 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
28/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Stale
Tech stack
python, sql
Domain
cli, documentation

Research direction

Start with the syntax-highlighting documentation linked in the issue and the Lexer abstract base class, then inspect the linked VqlLexer example and its lex_document error. Document a complete custom lexer example for an application-specific SQL derivative and explain the required interface so the example works.

Written by the indexing model from the issue text.

Description

While the documentation indicates that "it is also possible to create a custom lexer by implementing the Lexer abstract base class" (1), it doens't really explain how. I couldn't find any tutorial or really complete code example to illustrate how it works. Does anyone have any suggestion or example? I'm looking to creating a lexer from an application-specific derivative of SQL.

At this point, I wrote this code, but it returns the following error: AttributeError: type object 'VqlLexer' has no attribute 'lex_document'

Dominant language
Python
Stars
10.6k
Forks
815
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from prompt-toolkit/python-prompt-toolkit

All issues in prompt-toolkit/python-prompt-toolkit

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.