Xilem scene/multi-modal story

Nobody has claimed this yet.

I'm looking at Xilem and read through @raphlinus blog post on Xilem, and I'm trying to align my current thinking about the future of UI localization architecture in context of that research.

Due to my current role, I'm particularly interested in the future of multi-modal user interfaces. By that I mean not just GUI, but VUI interfaces and multi modal scenarios where VUI and GUI coexist and UI animations/transitions are kinetically driven by user gestures and animation phases.

To illustrate the space, I'll paint a user story from 2028:

> Anne walks into her bathroom, where a smart mirror dispays her day agenda, weather and other information. Upon entrance, the bathroom speaker greets her and enters a sequence of morning updates. The updates are complementary to what she sees on the mirror - for example since she does see the date, the VUI uses higher-context approach not repeating that information and utterances are assuming the context of the day. During the routine, she receives a call, which gets displayed on the screen and she can tap to respond. The audio provides additional cue and Voice Assistant asks if Anne would like to pick up the call. Anne responds that she's busy and asks for the Voice Assistant to schedule a call back in 10 minutes.
> Anne then walks to her closet, starts dressing up and her smart mirror provides information on weather today along her destinations. She asks via voice on what colors match best the blouse she holds in her hands and the mirror shows palette while the speaker responds with selection of tones.
> Anne presses a button on the closet mirror which enables music and starts moving it to an area which corresponds to the music that will be played in the kitchen during her breakfast. Her motion is fading-in the music as she moves it closer, but she pulls her finger and the button returns to original position and the music fades out.
> She then taps a button to start BBC daily news which triggers display of images and titles, but she turns away from the mirror, which the system detects and switches to read the news fully via VUI, since Anne is now not seeing any GUI in her vicinity.
> Anne then grabs her phone, which displays information on the screen, unfolds it partially which reorganizes the items on the screen to fit new available space, and flips one item up toward the direction of TV which triggers that item to show up on TV and start a video associated with that item of news.

Ok, this is quite futuristic, but there are several components that are pretty much within the range of UI development, and of particular interest to i18n/l10n.

First of all, we could think of this as a set of UIs, each with its own context, model and glue code, or we can reason that this is a single UI with multi-modal dynamically adjustable means of I/O. At times the UI is on smart mirror, phone and TV. The space on the phone dynamically changes as Anne folds/unfolds the phone, and the UI widget gets moved between Phone and TV while retaining state.

Second, Anne is moving around and the UI continuously adjusts to available means. At some point there's access to a speaker, mirror, phone and TV, but this can change rapidly and the UI adapts to that information.

Third, there's kinetic binding between UI elements, quite like Android UI has now, where the motion can be correlated with another scene change, visual or audible, and paused, slowed down, or sped up mid-motion which should trigger related parts of the scene to remain in sync.

Lastly, the UI is heavily contextual - depending on what modes are active, the same information is provided using different distribution (GUI only, GUI+VUI, VUI only, 2xGUI etc.) of means and allowing for different inputs.

============

I believe that we are moving toward such UX and prototyping architectures of UI should have an answer to such multi-modal scene design. Xilem seems to be very static-GUI oriented, and I'm wondering if that's a design limitation where everything above is out of scope, or a starting point for a conversation about UI architectures that can encompass such future directions?

Start by reading the Xilem discussion in this issue alongside the referenced blog post, focusing on how multi-modal input and output, changing device context, and synchronized scene transitions are described. Done would require a maintainer-defined direction on whether these scenarios are within Xilem's intended architecture or out of scope; the issue does not identify files or tests to change.

Written by the indexing model from the issue text.