Simple design model deformation
- Dominant language
- JavaScript
- Stars
- 15.7k
- Forks
- 3.9k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 34
Description
edit- the below design targets the `Primitive` API but it would be easier to target glTF `Model`s, and it seems like that may align better with our first use case anyway.
## Overview
We have some use cases for being able to edit design model geometry in Cesium. For now, the main use case involves editing models via a set of simplified control points. In computer graphics, this is often known as a deformer. There are many types of deformers (skeletal, lattice, curve-based, blend shapes, to name a few), and different bases that can be used to weight control points. For our initial use case, we're mainly talking about curve-based and plane-based deformations, like so:
Curve-based:

Plane-based:

In both cases, the underlying mesh has numerous vertices being deformed by a smaller set of control points.
## Goals
1. Ability to bind a `Deformer` (new) to one or more `Primitive`s and deform it by moving the control points.
1. Extensible method for moving control points (MVP: able to hook up to input fields or UI handles).
1. Ability to bake a deformation into the underlying mesh (and optionally delete the deformer after).
1. Flexibility in the deformer API for specifying how many control points (and where). (E.g. satisfies special case where control points _are_ a subset of mesh vertices for exact control over certain areas)
1. Performant: the initial binding (and user-triggered bakes) may be slow, but deformations should be hardware accelerated (in vertex shaders).
## Assumptions
1. The number of deformer control points is small compared to the number of vertices on the deformed mesh.
1. The number of control points that can be moved at once is relatively small.
Thus, it will be acceptable (from a performance standpoint) to represent and update the deformers' control points as `PointPrimitives` on the CPU (rather as a single custom `Primitive` with a custom shader to update positions on the GPU). This will allow for control points to easily participate in existing picking and styling workflows.
## Architecture
### Deformer API
Methods:
- `bind(primitive)` - do work to associate control points of deformer with the deformed mesh vertices.
- `unbind(primitive)` - discard binding.
- `bake(primitive)` - overwrite the original vertex positions with the deformed positions.
- `update()` - push the current state of the control points from the CPU to the GPU.
- `reset()` - (maybe?) restore control points to original state without discarding bindings.
- `setControlPoint(index, position)` - set the position of the control point at the given index.
- Optionally: life-cycle events (e.g. `onSetControlPoint`)
State:
- `controlPoints` - list (or other data structure) of control points.
- `bindings` - mapping of primitives to their binding.
Implementations of this interface will have constructors / options that determine on a per-implementation basis where and how many control points are created for a given mesh (and potentially other info like connectivity / adjacency of control points).
### GPU deformation
Each deformer implementation will own and maintain data that gets uploaded to the GPU (like vertex weights or rest / deformed control point positions).
The deformation itself will occur as part of the vertex shader step of a renderable. This is generally pretty simple: modify the rest-pose vertex position according to the current (relevant) control point positions and (possibly also) precomputed weights.
### User / Mouse Interaction
Deformer control points will be movable via mouse interaction. To determine which control point should be moved, we will perform scene picking on a mouse click. To avoid performing a pick for each deformer in the scene (among other benefits), we'll introduce a new widget: the `DeformerTool`
The `DeformerTool` widget will:
- Allow users to toggle the visibility (can I see the control points?), editability (can I move them?), and active state (am I seeing the rest pose or the deformed pose?) of deformers.
- Enforce that only one deformer is editable at a time.
- When editable: the deformer will listen to mouse clicks, perform picks, and drag control points.
The deformer tool also provides some foundation for future editing tools, should we venture that way.
### Lower level considerations / open questions
- Do we need access to the geometry data directly? For some types of deformers - no. We only need to be able to modify the shader. For other types of deformers, yes - we need vertex data to compute a binding. For this MVP, we will not need vertex data.
- How do we access geometry data? Generally speaking, once a renderable is created, its mesh data only exists on the GPU.
- How do we modify shaders? If we're targeting glTF models, this is actually easy - a pipeline stage (similar to `SkinningPipelineStage`). For primitives, we can completely override shader source but it's difficult to append or modify it the way glTF allows. This may require an enhancement if we want to target Primitives.
- How does the `DeformerTool` determine which object in the scene you want to toggle the deformer for? The last selected object? Do we then need some sort of outline highlighting to indicate that object? How does _that_ feature get turned on or off?
## Out of scope (maybe in the future)
1. Moving edges / faces of deformer geometry (will require something more complex than using a `PointPrimitiveCollection` as the CPU representation of the deformer).
1. Multiple deformers bound to a single primitive. (How to enforce this while keeping Primitive unaware of Deformer?)
1. Binding of deformers to other renderables (glTF models wouldn't be too hard. Terrain tiles and/or tilesets would be difficult but perhaps possible).
1. Animation of deformer control points.
1. General editable geometry (i.e. directly moving vertices, edges, and faces). (Having this would allow us to represent a deformer as editable geometry, itself, so that we could move its edges and faces - see first point)
Contributor guide
Assessment
This issue has not been assessed yet.