microsoft / microsoft/semantic-kernel

Python: CodeTokenizer splits quoted named-arg values on spaces (NamedArgBlockSyntaxError)

Open Beginner friendly
#14,460 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

python triage
Dominant language
C#
Stars
28.6k
Forks
4.8k
Avg merge
14h 13m
Merged PRs (30d)
18

Description

Description

CodeTokenizer splits named-argument quoted values at spaces, so a template like:

{{func key='hello world'}}

fails with NamedArgBlockSyntaxError: A NamedArgBlock starts with a name... — the token for the argument is cut to key='hello and parsing breaks. Any quoted named-arg value containing a space breaks the whole template.

Reproduction
from semantic_kernel.template_engine.code_tokenizer import CodeTokenizer
CodeTokenizer().tokenize("{{func key='hello world'}}")
# NamedArgBlockSyntaxError

Same with double quotes.

Expected behavior

quoted values are single tokens, spaces included, matching how the value grammar is documented (ValBlock is text surrounded by quotes).

Environment

semantic-kernel python main, Python 3.12

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CodeTokenizer in semantic_kernel/template_engine/code_tokenizer and run the provided tokenize reproduction for both single- and double-quoted values. Done means each quoted named-argument value containing spaces remains one token and the example templates tokenize without NamedArgBlockSyntaxError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.