a2ui-project / a2ui-project/a2ui

Ability to declare client-side event handling where one component can affect another component

Offen
#302 8 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
P2 status: needs-team-input status: needs-triage type: feature/enhancement
Vorherrschende Sprache
TypeScript
Sterne
16.4k
Forks
1.3k
Ø Merge
2 T. 13 Std.
Gemergte PRs (30 T.)
134

Beschreibung

Since I, in parallel, worked on a (much smaller) project that also returns agent-generated UI specs based on user prompts, I want to state the following proposal for discussion (and possibly refinement, if not outright-rejected). I purposely keep it relatively vague without going into implementation details in order to keep it short & the discussion on a high level initially, since it may introduce principle target use case incompatibilities with A2UI and thus cause a "desk reject", not warranting deeper investigation.

# Use Case

As a _developer_, I want to be able to enable users to "live" preview the results of their UI interactions without requiring re-prompting the LLM for efficiency and UX, at least for simple transformations/calculations.

As a _user_, I want to be able to make fine-adjustments within the generated UI and receive feedback on them without delays. E.g. in an LLM-based color manipulator.

```
1. Initial prompt: "dark red"
2. Output: Multi-choice selector with color previews of 3 "dark red" variants

3. Second prompt: "make it more saturated"
4. Output: Slider with saturation adjustment and live preview of saturation change of the chosen color (variant A) // <-- this requires either a specialized component, **or a generated callback function that is executed on the client-side**

```

# Current limitations

The current protocol is limited to pre-defined components with built-in business logic (e.g. color saturation change of color in the above example) limiting user-defined actions to strongly pre-defined set OR a whole server<->client roundtrip with A2A messages etc. introducing potentially significant delays, especially as soon as the LLM is re-involved.

# Extension Concept

1. Enable the (LLM) agent to generate expressions or functions with widget-specific parameters (e.g. slider min, max, position) by introducing the required protocol extensions, e.g. based on a python slang or pure math expressions
2. The functions are executed as a callback to the respective UI component's action (e.g. a slider and a slider position update) client-sided in a sandboxed logic interpreter / expression evaluator, such as python's `asteval` or `simpleeval` or typescript `filtrex` or similar
3. The respective UI component is updated purely client-sided with the function result (e.g. the color preview)

## Benefits

This allows for much more flexible UIs with instant, client-side-computed previews, etc. which are particularly important for fine-tuning results, such as small percentage adjustments in relative changes (for colors: hue, saturation, vibrance, ..) or non-conforming input formats (e.g. a color preview requiring hex input but the user providing rgba in floating point).

## Potential Risks
1. Security issues: expression/script evaluators come with the risk of suffering from security issues, e.g. `expr-eval` with [CVE-2025-12735](https://nvd.nist.gov/vuln/detail/CVE-2025-12735)
2. Incompatibility with the target use case of the A2UI protocol
3. ...?

---

I am looking forward to your feedback, hopefully a discussion and will try to answer all potential questions.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.