alecthomas / alecthomas/participle
Stable v2 release (API changes)
- Linguagem predominante
- Go
- Estrelas
- 3.9k
- Forks
- 213
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Descrição
Now that Participle has proven its initial concept, I think it's time to clean up the API. This will be a backwards incompatible change.
Work has started in the [v1 branch](https://github.com/alecthomas/participle/tree/v1).
- [x] Consolidate on Stateful lexer (1444519b50b541162cd4c2f34b36f037a06f0fed)
- [x] Optimise performance of the lexer (#111)
- [x] Make specifying filename explicit. This removes confusion and ambiguity. (cf6162a6162b85732482ecec972cf1b793c1c80c)
- [x] Get rid of unquoting hacks in text/scanner lexer. (4f53af941491a38b02c4ddd89c478c60529f1a8c)
- [x] Clean up error functions. (895f942ebdd09acb65769727864b36160eed35b0)
- [x] Eliminate internal unquoting and single quote munging from text/scanner based lexer. (4f53af941491a38b02c4ddd89c478c60529f1a8c)
- [x] Extend the concept of `Pos`/`EndPos` to support capturing the full range of tokens the node matched, including `Elide()`ed tokens. (2ace05e38cbb7b034da7dfb53cf22f893dcd3d89)
- [x] Refactor `Mapper` to eliminate the need for `DropToken`. (f82f61571f509811ffa0b970ab6f1ff49e5179d5)
- [x] Capture directly into fields of type `lexer.Token` and `[]lexer.Token`. (3b1f1514b50f2defd71f4197d8b44b654348274f)
Maybe:
- [ ] Extend `participle.Elide()` support so that elided tokens *can* be captured explicitly by name (but also see next point).
- [ ] Support streaming tokens from an `io.Reader` - currently the full input text is read.
- [ ] Refactor PeekingLexer so it doesn't consume all tokens up front.
Once the API is stable, some additional changes would be welcome:
- [ ] Optimise the parser.
- [x] Code generation for lexing (e2b420f4e9a6e6dd07ffac14bc02c4692aaff423).
- [ ] Code generation for parsing.
- [ ] Improve error reporting.
- [ ] Error tolerant parsing.
- [ ] LSP support? Can this be generalised?
- [ ] Generate syntax definition files for Textmate etc.?!
---
Regarding streaming, I'm not convinced this is a worth the considerable extra complexity it will add to the implementation. For comparison, pigeon also does not support streaming.
Additionally, to support the ability to capture raw tokens into the AST, participle will need to potentially buffer all tokens anyway, effectively eliminating the usefulness of streaming. It also vastly increases the complexity of the lexers, requiring three paths (`io.Reader`, `string` and `[]byte`), `PeekingLexer`, etc.
This increased complexity is mainly due to the lookahead branching, and the lexer needs to have a similar implementation to the rewinder RuneReader code (https://play.golang.org/p/uZQySClYrxR). This is because for each branch the state of the lexer has to be stored but also, additionally, as each branch progresses it needs to preserve any new tokens that are buffered so that if the branch is not accepted the parent can remain consistent.
There's also a non-trivial amount of overhead introduced for reading each token, as opposed to the current PeekingLexer which is just an array index.
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Avaliação
Esta issue ainda não foi avaliada.