alex / alex/rply

can't lex byte strings on Python 3

未关闭
#55 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
391
派生
62
PR 合并指标
30 天内没有已合并 PR

描述

I wanted a Python 3 lexer that consumes byte strings, but this doesn't seem possible with LexerGenerator. For example, for this test program:

``` python
from rply import LexerGenerator
lg = LexerGenerator()
lg.add('NUMBER', br'\d+')
lg.add('ADD', br'\+')
lg.ignore(br'\s+')
lexer = lg.build()
for token in lexer.lex(b'1 + 1'):
print(token)
```

you get:

```
Traceback (most recent call last):
File "test.py", line 7, in
for token in lexer.lex(b'1 + 1'):
File "/usr/lib/python3/dist-packages/rply/lexer.py", line 56, in __next__
return self.next()
File "/usr/lib/python3/dist-packages/rply/lexer.py", line 46, in next
colno = self._update_pos(match)
File "/usr/lib/python3/dist-packages/rply/lexer.py", line 27, in _update_pos
self._lineno += self.s.count("\n", match.start, match.end)
TypeError: a bytes-like object is required, not 'str'
```

(I ended up writing my own lexer for unrelated reasons, so this is not a show-stopper for me, but I thought you might want to fix it.)

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。