Defining custom cell handlers in Extension and interaction with computation engine

This came up through discussion in https://github.com/quarto-dev/quarto-cli/discussions/4761 by @coatless through the use case of webr Quarto extension (https://github.com/coatless/quarto-webr)

Main question is :

Is there a way to register engine requirements or options in a quarto extension, e.g. _extension.yaml?

Opening this issue to track idea and discussion. Below some thoughts on all this to consider (long writing to write all my thoughts for later reference)

---

webr extension syntax used is inspired by shinylive extension

---
title: "Testing"
format: html
filters:
- webr
---

```{webr}
1 + 1
```

When this is run with engine: knitr for example, there will be some interaction

• A warning

  Warning message:
  In get_engine(options$engine) :
    Unknown language engine 'webr' (must be registered via knit_engines$set()).
  

• A chunk processing by knitr which will lead to different output in intermediate .md file (because Quarto adds the .cell-code class for example to wrap knitr results .

  jupyter

   [trace] Parsed [CodeBlock ("",["{webr}"],[]) "1 + 1 "] at line 12
   [trace] Parsed [CodeBlock ("",["{webr}"],[]) "message(\"hello\")"] at line 17
  

  knitr

   [trace] Parsed [CodeBlock ("",["webr","cell-code"],[]) "message(\"hello\")" at line 20
   [trace] Parsed [Div ("",["cell"],[]) [CodeBlock ("",["webr","cell-code"],[] at line 23
  

We don't have those issue with our internal cell handler because we define a custom knitr engine for those and specific processing (no wrapping).

Behind this usecase, there is the generic question of How to use Quarto extension to define a CodeBlock handler in Lua and have it to play well with the rest ?

@jjallaire I know we discussed in the past the Quarto Extension for custom engine and it was not something straightfoward to consider because best integration would mean some typescript processing as we do for ojs, mermaid and dot. However, it seems there is a need to something, maybe just tooling, for Quarto extension that would define specific CodeBlock processing. Two use case for now shinylive and webr.

More thoughts on the current context based on my answer https://github.com/quarto-dev/quarto-cli/discussions/4761#discussioncomment-5319569

Using standard Pandoc fenced code attribute syntax prevent any processing by knitr. This means something like

```{.webr}
1 + 1
```

knitr will not parse and process this part of the code, and then Lua filter can target using el.attr.classes:includes("webr").
As notes shinylive uses shinylive-python as cell engine name. Using dash - makes knitr also not see that as a knitr engine to process.

This is an easy solution to make distinction between computation (or internal) cell handlers and Lua based ones.

Supports for YAML comments as option is a question though, and this is where shinylive solves this by using typescript for parsing comments too called from Lua directly

This shows that probably some tooling is necessary in Quarto Lua API if we want to help with this pattern.

OTOH, best would probably be to use common cell handler syntax like

```{webr}
1 + 1
```

This requires probably a way to indicate to computation engines like knitr to skip the parsing and processing of such cells - exactly what we do for ojs, mermaid and dot. Currently we tell knitr to output those cells as is.

Making that possible directly from extension configuration would allow to say to computation engine from the extension to pass through the code block.

Anyhow, some challenges are:

• What pattern do we want to promote for this ?
  • Cell handler be done exclusively from Lua as shinylive and webr as of now ?
  • or maybe some pre-processing defined by extension that would run as part of quarto render typescript pre processing ?
• What helpers tooling could be necessary in Quarto Lua API for extension developer ?
• How does this integrate and behave correctly with YAML intelligence for options as comments and existing computation engine, with their own cell engine ?