lark-parser / lark-parser/lark_cython

Fails to parse round-trip string objects from ruamel.yaml

Open
#36 4 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Cython
Stars
51
Forks
9
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

When parsing text from a YAML file in ruamel.YAML it it returns a `SingleQuotedScalarString` object, which does inherit from the `str` type. Sending this string to the pure-python lark parser seems to work fine, but when sending it to the cython variant it throws a TypeError.

**To Reproduce**

The following is a MWE that reproduces the issue:

```python
"""
Requirements:
pip install ruamel.yaml lark-cython lark
"""
import io
import ruamel.yaml

NEW_RUAMEL = 1

class _YamlRepresenter:

@staticmethod
def str_presenter(dumper, data):
# https://stackoverflow.com/questions/8640959/how-can-i-control-what-scalar-form-pyyaml-uses-for-my-data
if len(data.splitlines()) > 1 or '\n' in data:
text_list = [line.rstrip() for line in data.splitlines()]
fixed_data = '\n'.join(text_list)
return dumper.represent_scalar('tag:yaml.org,2002:str', fixed_data, style='|')
return dumper.represent_scalar('tag:yaml.org,2002:str', data)

def _custom_new_ruaml_yaml_obj():
"""
References:
https://stackoverflow.com/questions/59635900/ruamel-yaml-custom-commentedmapping-for-custom-tags
https://stackoverflow.com/questions/528281/how-can-i-include-a-yaml-file-inside-another
https://stackoverflow.com/questions/76870413/using-a-custom-loader-with-ruamel-yaml-0-15-0
"""

# make a new instance, although you could get the YAML
# instance from the constructor argument
class CustomConstructor(ruamel.yaml.constructor.RoundTripConstructor):
...

class CustomRepresenter(ruamel.yaml.representer.RoundTripRepresenter):
...

CustomRepresenter.add_representer(str, _YamlRepresenter.str_presenter)
yaml_obj = ruamel.yaml.YAML()
yaml_obj.Constructor = CustomConstructor
yaml_obj.Representer = CustomRepresenter
yaml_obj.preserve_quotes = True
yaml_obj.width = float('inf')
return yaml_obj

def codeblock(text):
"""
Create a block of text that preserves all newlines and relative indentation
"""
import textwrap
return textwrap.dedent(text).strip('\n')

# For common constructs see:
# https://github.com/lark-parser/lark/blob/master/lark/grammars/common.lark
RESOLUTION_GRAMMAR_PARTS = codeblock(
'''
// Resolution parts of the grammar.
magnitude: NUMBER

unit: WORD

numeric_unit: (magnitude WS* unit)
implicit_unit: unit

resolved_unit: numeric_unit | implicit_unit

%import common.NUMBER
%import common.WS
%import common.WORD
''')

RESOLVED_UNIT_GRAMMAR = codeblock(
r'''
// RESOLVED WINDOW GRAMMAR. Eg. 2GSD
?start: resolved_unit
''') + '\n' + RESOLUTION_GRAMMAR_PARTS

def main():
yaml_obj = _custom_new_ruaml_yaml_obj()
file = io.StringIO("{key: '1mGSD'}")
data = yaml_obj.load(file)
text = data['key']

# https://github.com/lark-parser/lark/blob/master/docs/_static/lark_cheatsheet.pdf
import lark
try:
import lark_cython
parser = lark.Lark(RESOLVED_UNIT_GRAMMAR, start='start', parser='lalr', _plugins=lark_cython.plugins)
except ImportError:
parser = lark.Lark(RESOLVED_UNIT_GRAMMAR, start='start', parser='lalr')

print(f'{type(text)=}')
print(f'{text.__class__.__mro__=}')

parser.parse(text)

if __name__ == '__main__':
"""
CommandLine:
python ~/code/lark_cython/tests/test_yaml.py
"""
main()
```

The type information it prints before it fails is:

```
type(text)=
text.__class__.__mro__=(, , , )
```

I've tested with versions:

```
3.11.9 (main, May 14 2024, 08:04:54) [GCC 12.2.0]
ruamel.yaml.__version__ = 0.18.6
lark.__version__ = 1.1.9
lark_cython.__version__ = 0.0.15
```

And

```
3.11.9 (main, May 13 2024, 14:03:39) [GCC 11.4.0]
ruamel.yaml.__version__ = 0.17.22
lark.__version__ = 1.1.7
lark_cython.__version__ = 0.0.15
```

My thought is that cython would handle a class that inherits from a `str`, but perhaps it doesn't I'm not sure if this can be fixed on the lark-cython side, but I figured it was worth reporting.

My current workarond is to do something like this:

```python
try:
tree = parser.parse(text)
except TypeError:
if isinstance(text, str) and type(text) is not str:
# We could be in a case where cython is failing to handle
# overloaded string types. Try casting to a regular str.
tree = parser.parse(str(text))
else:
raise
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided minimal reproduction and compare the pure-Python Lark parser with the lark_cython plugin parser using ruamel.yaml's SingleQuotedScalarString input. Trace the plugin parser entry point that receives text and verify that parsing the str subclass completes without TypeError while preserving behavior for ordinary strings.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.