microsoft / microsoft/terminal

Allow 3p suggestion sources

Open
#17,344 1 comment 1 reaction 0 assignees View on GitHub
Area-Extensibility Issue-Scenario Product-Terminal
Dominant language
C++
Stars
105k
Forks
9.6k
Avg merge
3d 17h
Merged PRs (30d)
29

Description

_from my whiteboard notes_

Right now, Terminal supports a couple built-in suggestion sources.

What if we allowed even more? What if we wanted arbitrary apps to be able to generate suggestions? Possible use cases:

* an app that synthesizes suggestions from your `bash_history`
* Something like Fig[^1] / [inshellisense](https://github.com/microsoft/inshellisense)
* an app that generates suggestions from commands.dev[^2]
* Some LLM thing that generates suggestions based off the buffer
* A not contrived example: A 3p app that wants to manage credentials / PATs, but not store them in the Terminal directly. Thereby avoiding putting them on the clipboard too.[^3]

How would we actually support this?

We could pretty easy use AppServiceConnections with apps that list themselves as Terminal extensions. Kinda like how we have apps that can say they're fragment extensions. We'd open an app service connection, pass them some state, then get a list of suggestions back. Yes, yes, ASC aren't great, but this kind of scenario is like, literally what they were made for.

We'd also need to make a couple changes to how the suggestions UI displays suggestions. Right now it's all done with actions, all at once. Then, we hand those actions over to the palette, and let it filter them. I've got a mind to do something else.
* What if all the suggestion sources were `IAsyncAction<>`s? We hand those asyncs to the suggestions control, then await them. When they're done handing back their actions, we put them into the suggestions list.
* Then, what if we didn't even have the suggestions control do the filtering itself? What if we passed the filter text to the extension?
- That would help support LLM scenarios, so users could type "how do I foo the frob" and the suggestion would be "`foo --param frob.txt`"
- We'd have to somehow be able to apply a weighting/ordering to these results uniformly. We could let extension authors give us that weight, but then it'll be a weighting arms race.

Lastly, permissions.

Similar to what I wrote in https://github.com/microsoft/terminal/blob/dev/migrie/s/ai-providers/doc/specs/drafts/LLM/Terminal-AI-Extensions.md

> We'll need to present to users a number of settings to control how much context plugins are able to receive from the Terminal.
>
> * Currently selected text
> * Currently typed commandline
> * Most recent (N) command(s)
> * and errorlevels
> * and output
> * Environment variables
> * profile commandline(?) (we may want to always provide the target exe, without args, as a bare min)
> * Other panes too?

and we (terminal) need to store which things the user allowed in a _separate_, secure location. If the user disables permissions to a particular piece of data, then we need to either give the extension null (to explicitly indicate that it was rejected) or garbage data (the "ron swanson" option, don't even let the extension know that the permission was rejected)

-------------

Overall, it's a lot of work, and needs a long spec.

xlinks:

* #4000
* #3121

[^1]: RIP
[^2]: sure this is contrived, because you can just do https://github.com/microsoft/terminal/blob/dev/migrie/s/snippets/doc/specs/%231595%20-%20Suggestions%20UI/dump-workflows.py, but the point stands
[^3]: this was specifically called out in a mail thread "Windows Terminal extensions?" from 5/13/2024

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked issues #4000 and #3121, the Suggestions UI specification, and Terminal-AI-Extensions.md. The issue calls for a long specification covering extension connections, asynchronous suggestions, filtering and ordering, context permissions, and secure permission storage; done means those requirements are resolved in an agreed design.

Written by the indexing model from the issue text.

Assessment

Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.