Agentic UI

LLMs can generate UIs, we know this because the LocalAGI and LocalAI UIs are mostly generated by LLMs. However this was done with a lot of human guidance and the resulting code is fairly horrible (the functionality is great though :-D).

It would be incredible if we enabled agents to dynamically and reliably generate simple UIs or dashboards in response to user requests. For example if the user asks what the exchange rate is between USD and EUR, then the agent could create a simple form with input boxes for USD and EUR and a chart. Even simpler, it could provide drop-down boxes or buttons during chat similar to Slack apps.

Ultimately when the user is configuring their agents in LocalAGI, models in LocalAI, vector store in LocalRecall etc., we can generate settings pages with just the needed details based on what the user wants to do using a built-in settings agent. This would bring us full circle.

Essentially we want the agent to be able to create a micro app, there are a number of things that need to come together for this:

• Sandboxed code for performing calculations and actions
• Set of high level UI components available across all Local* apps
• Declarative way of specifying the UI
• Action selection for e.g. button actions

LocalAGI already has a lot of settings pages and to reduce the amount of frontend code, we have the backend specify the form fields required using a meta-data format. We can build on this to provide a set of components for all our settings pages and then also UIs generated by AI.

A typical HTML/Javascript frontend framework is far too complex and open-ended for these purposes. Qt QML is also too complex, but is going in the right direction. We want something that is declarative and easily represented as JSON/YAML with minimal layout information. To avoid presenting the user with a broken UI, the UI definition should be amenable to static analysis.

The (essential) components should be commonly available in native component libraries, so that the web frontend can be substituted for a native GUI.

Related: #180 (The Agent-User Interaction Protocol), AFAICT this just helps display the agent's status, you can't specify buttons for e.g.

No files or tests are named. Start by reading LocalAGI's existing settings-page metadata implementation and related issue #180, then map the proposed sandbox, shared components, declarative schema, and action handling. Done would require an agreed, statically analyzable UI format that supports the described interactive micro-apps.