screenshots, diffs to annotate scrollback/history of fullscreen (tui/curses) apps

Start by reviewing Warp's current Data model and the referenced alacritty_terminal Grid serialization and deserialization APIs. Trace how each fullscreen app block's grid is handled at precmd time. The issue is not complete until the capture representation, trigger timing, and annotation behavior are specified.

from #78

I was just using htop for the first time in a while, and I had a thought that it would be interesting to extend (or merely apply) the notion of excerpting like I describe in #23 (3) to such apps (fullscreen / TUI / curses).

Status Quo

Currently when you use such an app (htop, vim, less, etc.) in Warp, once you exit it - the history block is rendered showing only the command

This is of very limited use to have in either the immediate scrollback of an active session, or in a retained artifact of a past session

(either: as I imagine in #23, something where you save a (shared) session to use as part of an incident post-mortem learning review or perhaps an educational tutorial / screencast, akin to https://asciinema.org et al ; or perhaps in the persistent cloud history that I believe is more solidly on the roadmap #33 )

What would be more useful?

What comes to my mind is some concept of capturing one or more "screenshots" of the app as it's running. Another idea (more suitable for editors) is a diff of a file before and after invocation (at least for file(s) which are passed as command line argument(s) to an editor, rather than opened from within it).

Mocking that up, I imagine something like:
screenshots-mockup.psd.zip

(For less and other pagers, it might also just be good to include a scrollable fixed-height buffer of the file contents; very much like what is currently shown in the share modal; or just a file:/// hyperlink?)

How?

These "screenshot"s could either:

1. render the contents of the ~frame buffer captured as it appeared on screen- capturing text as text (so copy and pasteable, etc.), ansi escapes as is (raw or post processed);

   this might be easy enough with the current Data model, since I see that https://docs.rs/alacritty_terminal/0.14.0/alacritty_terminal/grid/struct.Grid.html#impl-Serialize and https://docs.rs/alacritty_terminal/0.14.0/alacritty_terminal/grid/struct.Grid.html#impl-Deserialize%3C%27de%3E

   this would creating a richer, likely smaller and more compressible artifact – and one more alike to what is currently offered with sharable command output; but could require more work

   and*/or

2. capture a raw rasterized screenshot of the window (in macOS, at least, I believe that capturing your own window this way does not require screen recording permission, but I might be wrong)- probably as a png.

*it likely would make sense to do (1) but e.g. have rasterized thumbnails

When?

It might make sense to make such capture at one or more of:

1. Application exit time. I think of this like what you see in the iOS multi-tasking view.

   Since each block already has its own grid that is being discarded(?) at precmd time, you would just need to serialize rather than discard that (as I guess you already do for non-fullscreen apps)

2. Periodically. I think of this like macOS's Time Machine visualization. This would be triggered either by some clock time interval, or heuristics like:

   • snooping on file saves (fd flushes of the exec'd process? or just plain old File System Events API / inotify ; for at least files mentioned in invoking command?)
   • a heuristic on how much of the screen is redrawn at once (between keystrokes)
   • network traffic?

3. (Only or additionally) by user request—via command palette, keyboard shortcut

   or maybe even a button-on-hover; possibly a menu option under a version of the ⋮ like we see in scrollback blocks, but just floated in upper right while a fullscreen app is active.

   When thus HCI-initiated it might make sense also to prompt for a caption/label (to indicate what is intended to be highlighted/captured), crop(?), and maybe even to (at screenshot time or after-the-fact) allow further annotations
   (something akin to, or identically (if rasterizing): macOS markup / Skitch / CleanShotX's image editor )