Dimos Web Proposal

A base overview and some issues we need to decide on. I'll give the pros and cons for each alternative.

Base overview

I'm calling the browser app Dimos Cockpit to have a clear term.

• The user opens Dimos Cockpit in a browser. It would be TypeScript app with React and Three.js. You can view lidar and camera data. Teleop with keyboard controls. In the future you should be able to talk to the agent or even to view dtop performance. A single interface for everything.
• dimos runs on the robot controller on the LAN, next to the robot (Unitree Go2, LYNX M20).
• The broker is a small web service to connect robot controllers (dimos app) to the Cockpit. You register using the --broker-url argument: dimos run --broker-url URL.
• Every link is two-way: media and telemetry up, control down.

1. How much does the frontend see?

This is the main question.

First, the fat frontend alternative. With this, the Cockpit is in control of requesting the topics it needs and processing them how it wants.

Some are too intense (e.g. lidar) so we would have decimators on the backend to make them usable on the frontend.

This makes the Cockpit quite flexible and it's not tied to what the backend does.

But certain transformations aren't even possible. For example, for lidar stuff, that requires specialized libraries which don't exist for the web.

The alternative is the thin frontend. In this case, we'd have a module which processes all the data in a way that's suitable for the Cockpit views.

This makes it less flexible. The Cockpit and the "Cockpit adapter module" on the backend are deeply coupled. But the advantage is that we only send to the frontend what it needs and what it can easily process.

A single JS thread is not good at processing large amounts of data. When we were using Foxglove we encountered this issue. It's much better to process in the backend because we can scale the backend, but we can't scale people's browsers much. So this alternative will perform much better.

2. Where does the bridge run?

When we convert messages between DimOS and what the broker recieves, where should we do that. Let's call that the bridge (maybe we need a better term).

I can see two alternatives:

• The bridge is a DimOS Module. It listens to topics and processes them and sends them to the broker
• The bridge is a separate process. It can be python that listens to LCM/Zenoh, or it could be a Node app, Demo, bun, Rust, etc.

The advantage of doing this sort of stuff in TypeScript is that you can reuse some of the same code in the Cockpit.

(BTW, this is why I'm using the term Cockpit for what runs in the browser. People often use the term frontend for the backend app which talks to the frontend)

3. Cloud and LAN?

So far we're considering running the broker in the cloud. But it would be nice to run the whole thing locally as well.

It's very useful to use the same Cockpit for controlling your robot over LAN. In this case a proposal would be to run dimos run --local-broker instead of dimos run --broker URL and you'd get a local broker and local Cockpit.

(This would also make development easier.)

4. How to transport?

Do we go with WebSockets, WebRTC, or WebTransport?

WebTransport seems like the clear winner because it's built for multiple streams, low-latency, and you can choose
between high reliabilility and low reliability messages.

No implementation files or tests are named. Start by reviewing the four open architecture questions: frontend processing, bridge placement, cloud versus LAN deployment, and transport; done means the alternatives are evaluated and a concrete Cockpit, broker, and bridge design is documented.