support for json (or other?) grammar?

llama.cpp now supports grammars:

https://til.simonwillison.net/llms/llama-cpp-python-grammars

Is that something that will come to candle?

It sounds like the approach taken in this python library would be straight forward:

https://github.com/1rgs/jsonformer/blob/main/jsonformer/main.py

Basically, since you know the JSON schema, you return appropriate LLM tokens for structure based on control flow, and constrain logit output for typed value situations.

I started to work on this approach in a demo codebase... I'll report back on any progress.

Curious to hear from others about how feasible the approach is.

Start by reading the linked llama.cpp grammar material and jsonformer/main.py to understand the proposed constrained-generation approach. The issue does not name a Candle entry point or tests; progress would require defining a concrete scope for JSON or other grammars, identifying where it belongs, and establishing tests for the supported behavior.

Written by the indexing model from the issue text.