facebook / facebook/astryx

feat: Native platform support — Android components for smart glasses & mobile

Open
#1,190 0 comments 0 reactions 0 assignees View on GitHub
discussion enhancement theming
Dominant language
TypeScript
Stars
13.1k
Forks
1.1k
Avg merge
1d 14h
Merged PRs (30d)
669

Description

## Summary

XDS currently targets web (React + StyleX). This proposal extends XDS to **native platforms** — starting with Android (Kotlin/Compose) for Meta smart glasses (Ray-Ban Meta) and the companion mobile app.

The key insight: XDS already has a clean separation between **tokens**, **theme definitions**, and **component implementations**. The `defineTheme()` → `xds theme build` pipeline resolves all tokens and component style overrides into a structured intermediate format. Today that format generates CSS. It can also generate **Kotlin**, **Swift**, or any other platform's style definitions — from the same theme source.

## Motivation

Meta smart glasses run a native Android UI on a **600×600 square display** rendered through a see-through prism. The current SystemUI uses a Panels system with Bloks-based rendering — native Android views inflated from serialized payloads sent by the companion phone.

There's no web rendering surface on the glasses today. But there *is* a need for:
1. A consistent design language across glasses experiences
2. A theming system that works on constrained displays
3. Components purpose-built for the interaction model (head gestures, voice, no touch)

XDS can provide all three — if it speaks native.

## Two Distinct Surfaces

Smart glasses and the companion mobile app are **fundamentally different surfaces** and should be treated as separate targets:

### Glasses

| Constraint | Value |
|---|---|
| Display | 600×600px square, see-through prism |
| Color mode | Always dark (transparent background, light-on-dark text) |
| Interaction | Head gestures, voice commands, touchpad on temple — no touchscreen |
| Typography | Larger base sizes for readability at arm's length through optics |
| Motion | Minimal — battery constraints + perceptual limits of see-through display |
| Component set | Reduced: cards, text, icons, lists, status indicators, buttons. No dialogs, popovers, tables, forms |
| Layout | Card-based panels, single-focus UI, no scrolling |

### Mobile Companion

| Constraint | Value |
|---|---|
| Display | Standard phone screen, full color, light + dark mode |
| Interaction | Standard touch, gestures, keyboard |
| Typography | Standard mobile sizes |
| Motion | Full motion system |
| Component set | Broader — closer to web, but adapted for mobile patterns |
| Layout | Standard mobile layouts, navigation patterns |

These might share a base Android component library but diverge significantly in theme defaults, component subsets, and interaction patterns.

## Architecture

### Theme Build Targets

`xds theme build` gains `--target` flag:

```bash
xds theme build src/glassesTheme.ts # CSS (default)
xds theme build src/glassesTheme.ts --target kotlin # → XDSGlassesTheme.kt
xds theme build src/glassesTheme.ts --target swift # → XDSGlassesTheme.swift (future)
```

### What Gets Generated

Not just tokens — **full component style definitions**. The resolved theme already contains:

- `theme.tokens` → all resolved values (type scale, motion, radii expanded)
- `theme.components[name].base` → base component styles
- `theme.components[name]['variant:primary']` → variant styles
- `theme.components[name]['variant:secondary+size:sm']` → compound variant styles

The Kotlin generator maps CSS properties to native equivalents:

| CSS Property | Kotlin/Compose |
|---|---|
| `backgroundColor` | `Color(0xFF...)` |
| `borderRadius` | `RoundedCornerShape(X.dp)` |
| `fontSize` | `X.sp` |
| `padding*` | `PaddingValues(...)` |
| `fontWeight` | `FontWeight.Medium` |
| `color` | `Color(0xFF...)` |
| `gap` | `Arrangement.spacedBy(X.dp)` |

Generated output example:

```kotlin
// Generated by xds theme build --target kotlin
object XDSGlassesTheme : XDSTheme {
// Tokens
override val colorAccent = Color(0xFF4FC3F7)
override val colorTextPrimary = Color(0xFFFFFFFF)
override val spacingOuter = 12.dp
override val radiusContainer = 16.dp

// Component styles
override val button = ButtonStyles(
base = ButtonStyle(
cornerRadius = 8.dp,
paddingHorizontal = 12.dp,
paddingVertical = 8.dp,
fontSize = 14.sp,
fontWeight = FontWeight.Medium,
),
primary = ButtonStyle(
backgroundColor = Color(0xFF0064E0),
contentColor = Color(0xFFFFFFFF),
),
secondary = ButtonStyle(
backgroundColor = Color(0x1A053659),
),
)
}
```

### Package Structure

```
packages/
core/ ← React + StyleX (web) — existing
android/ ← Kotlin/Compose component library
src/
components/ ← XDSButton, XDSCard, XDSText, XDSStack...
theme/ ← XDSTheme interface, CompositionLocal provider
themes/
default/ ← builds to CSS + Kotlin + Swift
glasses/ ← NEW: always-dark, high-contrast, large targets
```

### Native Components

The native packages ship **behavior + layout + accessibility**. Visual styling comes entirely from the generated theme:

```kotlin
@Composable
fun XDSButton(
label: String,
variant: ButtonVariant = ButtonVariant.Primary,
size: ButtonSize = ButtonSize.Medium,
onClick: () -> Unit,
isDisabled: Boolean = false,
isLoading: Boolean = false,
icon: @Composable (() -> Unit)? = null,
) {
val theme = LocalXDSTheme.current
val style = theme.button.resolve(variant, size)
// Component handles layout, interaction, a11y
// Style object handles all visual properties
}
```

Same API conventions as `@xds/core` — same prop names, same variants, same mental model.

## Glasses-Specific Theme

```typescript
const glassesTheme = defineTheme({
name: 'glasses',
typography: { scale: { base: 18, ratio: 1.15 } },
motion: { fast: 100, medium: 250, ratio: 0.8 },
tokens: {
'--color-background-page': '#000000',
'--color-background-surface': 'rgba(20,20,20,0.85)',
'--color-text-primary': '#FFFFFF',
'--color-accent': '#4FC3F7',
'--radius-container': '16px',
},
components: {
button: {
base: { minHeight: '48px' },
},
card: {
base: {
backgroundColor: 'rgba(30,30,30,0.9)',
},
},
},
});
```

## Open Questions

1. **Package naming**: `@xds/android` vs `@xds/android-glasses` vs `@xds/glasses`? Should glasses be a subset of a broader Android package, or its own thing?
2. **Bloks integration**: The glasses Panels system uses Bloks for rendering. Should XDS generate Bloks payloads directly, or target Compose and let the platform bridge handle it?
3. **Component subset**: What's the minimum viable component set for glasses? Likely: Text, Button, Card, Icon, Stack, List, StatusDot, Spinner, Badge. What else?
4. **Interaction model**: How do head gestures and voice map to component interactions? Does Button need a `gazeTarget` prop? Does List need gesture-based navigation?
5. **iOS/SwiftUI**: The companion app exists on iOS too. When does `@xds/ios` enter the picture?
6. **Display fidelity**: The see-through prism has different color reproduction than a standard screen. Do we need glasses-specific color calibration in the theme?

## Prior Art

- The Panels system (`com.meta.wearable.panels`) already uses a widget-based architecture with data providers, which maps naturally to component + theme
- The existing SystemUI at `fbandroid/java/com/meta/smartglass/app/systemui/launcher/` uses a `PanelsManager` with `BloksRenderer` for glanceable widgets
- The emulator is available via `maui nova e -t greatwhite` (Hypernova) — 600×600 display, testable without physical hardware

## Phases

**Phase 1: Token + Style Generation**
- Add `--target kotlin` to `xds theme build`
- Generate complete token objects + component style definitions from the resolved theme
- Create `@xds/theme-glasses` with glasses-specific defaults

**Phase 2: Android Component Library**
- `@xds/android` package with Compose components
- XDSTheme provider consuming generated theme files
- Core components: Text, Button, Card, Icon, Stack

**Phase 3: Glasses Integration**
- Glasses-specific component adaptations (gesture targets, voice triggers)
- Bloks payload generation or Compose-to-Bloks bridge
- Emulator testing pipeline

**Phase 4: Multi-Platform Parity**
- Swift/SwiftUI target for iOS companion
- Cross-platform component API alignment
- Shared storybook/catalog for all platforms
---

## Exploration Notes (from prototyping session)

We built a working prototype: XDS components running on the Hypernova smart glasses emulator (600×600 display). This section captures ideas and findings from that session.

### Additive Light Display

Black pixels = transparent on the real glasses (the display is a projector onto a see-through prism). This fundamentally changes color theory:

- **You can only add light, never darken.** No shadows, no overlays that dim.
- **Bright colors (red, green, yellow) project strongly** through the prism
- **Blues are perceptually weaker** — shorter wavelength, less visible
- **White is the most "opaque" thing you can render**
- **Dark grays are nearly invisible** against bright environments

This means `light-dark()` doesn't apply. There's no light mode when black = transparent. But the existing `[a, b]` tuple format in `defineTheme` maps perfectly to **`[bright environment, dark environment]`** — same data shape, different semantic. The ambient light sensor drives which side of the tuple to use instead of `prefers-color-scheme`.

### Ambient-Adaptive Theming

The glasses have an ambient light sensor (`OPT3001`, 13-step lux-to-nits curve from 0 to 10,000 lux). We prototyped adaptive theming where the full visual treatment shifts based on environment:

| Environment | Surfaces | Text | Accent |
|---|---|---|---|
| Dark room | Subtle `rgba(40,40,45,0.85)` | Soft white `#E8E8E8` | Cool blue `#4FC3F7` |
| Indoor | Medium brightness | Crisp white | Standard blue |
| Outdoor | Bright `rgba(80,80,85,0.9)` | Full white `#FFFFFF` | **Warm orange `#FFA726`** |

Orange/amber accent in bright conditions because warm colors have higher additive visibility through the prism than cool blues.

The `AdaptiveColor(bright, dark)` type interpolates smoothly between the two values based on a normalized ambient level (0.0 = dark room, 1.0 = bright sun), using logarithmic scaling of lux readings.

### Camera-Aware Environment Adaptation (Future)

Beyond brightness — the outward-facing cameras could inform the theme:

- **Complementary colors** — in a green park, shift accent to colors that pop against green
- **Clash avoidance** — if looking at a red wall, don't use red for errors (it'll blend in)
- **Background-aware contrast** — boost surface opacity where the background behind the UI is bright, keep subtle where it's dark
- **Time-of-day warmth** — golden hour → warm amber UI, overcast → cooler tones

The token system supports this naturally. `AdaptiveColor` could become a function of a richer environment signal:

```ts
interface GlassesEnvironment {
brightness: number; // 0-1 from light sensor
dominantHue: number; // 0-360 from camera
dominantSaturation: number; // 0-1
backgroundLuminance: number; // what's behind the UI region
}
```

This is only possible on glasses — no other display has a camera showing what's *behind* the screen.

### Architecture: Compile-Time vs Runtime

We explored two approaches to getting XDS on glasses:

#### Approach A: JSX → Kotlin Compiler

The developer writes standard XDS React. A compiler parses the JSX and emits native Kotlin:

```
GlassesApp.tsx (36 lines) → compile.mjs → MainActivity.kt (244 lines) → APK
```

**Built a working prototype.** The compiler uses Babel to parse JSX, walks the component tree, and emits Android Views with theme tokens baked in. We deployed it to the glasses emulator successfully.

Pros: No runtime overhead, tree-shaking, static analysis, tiny APK
Cons: Can't update UI without rebuilding, no personalization

#### Approach B: RSC-over-Bluetooth Runtime

React Server Components already serialize UI as a JSON payload. The companion phone renders React, sends the RSC payload over Bluetooth, and a native XDS runtime on the glasses inflates it:

```
Phone (React server) → RSC payload (Bluetooth) → Glasses (native XDS renderer)
```

This is architecturally identical to how the existing Bloks/Panels system works — just a different wire format.

**RSC vs Bloks comparison** (same UI, our demo screen):

| Dimension | RSC + XDS | Bloks |
|---|---|---|
| Payload size | ~1.1 KB | ~4.2 KB (JSON) / ~2.5 KB (protobuf) |
| Node count | 17 (semantic) | 47 (primitive) |
| Abstraction | `XDSCard` | `LinearLayout > RoundedRect > Text` |
| Theme resolution | On device (adaptive) | On phone (static) |
| Ambient adaptation | Zero BT traffic | Full panel re-render over BT |
| Update granularity | Single component | Full panel |

**The killer advantage: adaptive theming.** Bloks bakes every color into the payload on the phone. If ambient light changes, the phone re-renders and re-sends the entire panel. RSC sends `type: "XDSCard"` and the glasses resolve colors locally. Zero Bluetooth traffic for theme changes.

### Component Availability: Style-Only vs Behavioral

We analyzed all ~72 XDS components by checking for `useState`, form elements, interactive ARIA roles, and keyboard handling:

- **~47 style-only** (pure presentation) — Card, Badge, Text, Icon, Stack, Grid, List, Spinner, ProgressBar, etc. These compile mechanically from props to native styles.
- **~25 behavioral** (have state, keyboard handling, ARIA) — Button, Calendar, Selector, TextInput, Slider, TabList, etc. These need native implementations.

For glasses specifically, the behavioral set that matters is small: Button (gaze/voice target), List (gesture navigation), and possibly SegmentedControl.

### Input-Aware Component Tiers

The available input method determines which components are viable. With the neural wristband (bracelet), many "unsupported" components become usable:

| Input | Components Enabled |
|---|---|
| Voice + head gestures only | Card, Text, Badge, Button, simple List, StatusDot |
| + Bracelet (EMG/pinch) | Table, CodeBlock, 2D ScrollArea, Selector, long Lists, code review |
| + Keyboard projection | TextInput, TextArea, PowerSearch, Typeahead |

A code review experience on glasses sounds impossible with just voice — but with the bracelet providing pinch-to-scroll and tap-to-select, it becomes natural. The component manifest could be input-aware:

```ts
export const glassesTarget = {
core: ['XDSCard', 'XDSText', 'XDSBadge', 'XDSButton', 'XDSList'],
withBracelet: ['XDSTable', 'XDSCodeBlock', 'XDSScrollArea', 'XDSSelector'],
withKeyboard: ['XDSTextInput', 'XDSTextArea', 'XDSPowerSearch'],
};
```

### Working Prototype

The prototype lives at `~/xds/glasses-demo/` and `~/xds/glasses-compiler/`:

- `glasses-compiler/GlassesApp.tsx` — XDS React source (36 lines)
- `glasses-compiler/compile.mjs` — JSX → Kotlin compiler
- `glasses-demo/glassesTheme.ts` — XDS theme with `[bright, dark]` tuples
- `glasses-demo/` — Android project, builds and runs on Hypernova emulator

Setup: `maui nova e -t greatwhite` → `maui skoobe` → `adb install`

### Revised Phases

**Phase 1: Theme Build Targets** (unchanged)
- `--target kotlin` / `--target swift` for `xds theme build`
- `@xds/theme-glasses` with adaptive `[bright, dark]` tokens

**Phase 2: XDS → Native Compiler**
- Compile XDS JSX to native Kotlin (Android Views or Compose)
- Style-only components → generated code
- Behavioral components → imported from `@xds/android-glasses`
- Input-aware manifest for component availability

**Phase 3: RSC Runtime**
- Native XDS renderer on glasses (inflate RSC payloads)
- RSC-over-Bluetooth from companion phone
- Device-local theme resolution (ambient-adaptive, zero BT traffic)
- Compare performance vs Bloks on real hardware

**Phase 4: Environment-Aware Theming**
- Camera-informed color adaptation
- Background-luminance-aware contrast
- Complementary/clash-avoidance color selection
- Bracelet input integration for component tier unlocking

Contributor guide

Open the contributing guide

Research direction

This proposal spans the existing packages/core, the xds theme build entry point, and the Android SystemUI entry point at fbandroid/java/com/meta/smartglass/app/systemui/launcher/. Start by narrowing the work to one phase and reading the relevant theme-generation or emulator flow; done requires a defined scope, implementation target, and validation command, none of which is selected here.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, kotlin, react, typescript
Domain
design, mobile, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.