microsoft / microsoft/semantic-kernel
Python: CodeTokenizer splits quoted named-arg values on spaces (NamedArgBlockSyntaxError)
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 28.6k
- Forks
- 4.8k
- Avg merge
- 14h 13m
- Merged PRs (30d)
- 18
Description
Description
CodeTokenizer splits named-argument quoted values at spaces, so a template like:
{{func key='hello world'}}
fails with NamedArgBlockSyntaxError: A NamedArgBlock starts with a name... — the token for the argument is cut to key='hello and parsing breaks. Any quoted named-arg value containing a space breaks the whole template.
Reproduction
from semantic_kernel.template_engine.code_tokenizer import CodeTokenizer
CodeTokenizer().tokenize("{{func key='hello world'}}")
# NamedArgBlockSyntaxError
Same with double quotes.
Expected behavior
quoted values are single tokens, spaces included, matching how the value grammar is documented (ValBlock is text surrounded by quotes).
Environment
semantic-kernel python main, Python 3.12
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with CodeTokenizer in semantic_kernel/template_engine/code_tokenizer and run the provided tokenize reproduction for both single- and double-quoted values. Done means each quoted named-argument value containing spaces remains one token and the example templates tokenize without NamedArgBlockSyntaxError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100