CesiumGS / CesiumGS/cesium

Animation + CustomShader ergonomics

Open
#10,326 1 comment 0 reactions 0 assignees View on GitHub
category - model/gltf
Dominant language
JavaScript
Stars
15.8k
Forks
3.9k
Avg merge
4d 6h
Merged PRs (30d)
34

Description

What level of support should we have for animations in CustomShader? Should we add more struct fields for ease of use in CustomShader? Or should it be up to the glTF asset author to provide texture coordinates/metadata that are designed with procedural texturing in mind?

While testing #10314, I tried making [this local sandcastle](http://localhost:8080/Apps/Sandcastle/index.html#c=xVf/T+s2EP9XrP6UTuCkX2ih46GhwmNMVGO0epO2Tu+5idtYOHZkOwXexP++s52kacubhhBvVdXa5/N97nN3di6xFNqgNaMPVKEPSNAHNKaaFRn+5GTBvBW7+VgKQ5igat46QH/PBUI6lQVPzgXLiKEjZFRBD+biuf3jXMyFt4hjLuN7HBdKUWFmLKOAUdr/peCMiAvYi5dKZtdaHg+iTmAtz1vdqNs57HQOo5NZNxp1h6PeEEedQTTo94dHvf7x4Lh/0hv+MW/NhceLHY+40EZm05Qku2zGjZXAub9UZJWBV142o49mhL7YhbVkSb06Ac7Bx3JyLfLCoKV2/weICQnT+Gv2OZMJ5ROgohjhKCsHbR8nVAtwwpbLQtsggP3YBKUpTIxRbFEYqnEuNTNMisnYEkPo2f58qeMahmhiwXRF2UFrsPinVSwBjWMzb5XJSdAM4MWKU0ie14C9lNMYtJaFiC0gCmp/Ibdu0SEFlczmJZzmNNYhZI2E3o/wipvljbQxDCu4Ci1c8dnHPSlewY7aEfs5qoaeMpB2i/53h9KUZTmnEASVp+/LxiM5IE+kIXgbhzotzhYaF4vvlBiHZ+G2M1OLX0nrv1FDwW8FEYZ9pUn7/6B5WMO/mfDLtTi9Z+J7lKLFaVainb+NgL8c0YS82X+MQ/hOifXLkfD3UugRAKAcfYYh+LzYcjl6hcsXcHFKNJZZrqjWrtAA6L3cd2gbsIoO4/czVcCTbS/6r6MiBX3fwDuI5ngv9p3X1MuVkoVI0Ceaspi/m+sepQTZnr2pdK4FPIZzyYnzcEa1eYdTuwViMfYleyT+JQV/NbqbNN9pan6mJGFidctMnN5J7n0r16AbSbGRd6BBhA46x1HbGY38r++bvN0le6QJ9DgZnSnQXUqVbdq0WqQx9HKEez35sd5zRaEpJEaqsnUTUpnqyTxvPbgoV11aHdxmQAvFD1BK2So1Zci9W1UntPFlTBS0SMCn51rGC7pSlOoyI4edbg9Hw35/0DkpY9vv4+go6g2jQSnwKHbs/UFl54t1DCxwrlgGkGtowxTN5JqeQ0x9ThrNFvjzrV0kSUpvqizYDZePObR/toGEFtA6bm/2oC4ue4eMkI1CJanaScUeRy/kId1JezMbjfKs4tcoNKigxqy0fMk5yzU0vfj3q+lxv6HwQmHUlVqrNXvu0dZsw4fBc7fIbsEcn8KDeIQGG5iMPNrFKdQWLHQj+JRrz+1mplxQIC8kebqFGDJNsUmpCDYH1mnUp7Z6AwHvFYHalffnpo7OXmy+daqgY23GNNoP385BO+wctRtK3uuFvcLA5jRPqaJYgW6h0Q+Ng+8HFddqH7wjQF35rgW8dQVma7KuHaCVj7aKrVa+gSV8d3l7eT6r4+nfKOzfswWaEpHERBu4V8HwTEq+IGpCReFDqb07rYPWqTZPnJ5VqD9B+wGn3NZsADe4oXCDQ1ulwwU8E6nBsdbVTXYaNreeJmyNWPLhhZdKFHOiNawsC+6KZN46Ow1Bf28rly4/v66p4uTJqqWdsxsvxBifhjB9eafxDHcs/wM) which just colors the model by `positionMC` to make sure that works and it does. By the time the custom shader is invoked, `positionMC` contains the final position, after applying any morph targets and skins.

In some cases, this may be useful, if you want to color based on the current position in model space. However, if one was procedurally texturing a character, the original position before skinning might be more useful.

For example, the `CesiumMan` can't be easily textured using `positionMC` since this is the animated, skinned position, not the A-pose position.
![image](https://user-images.githubusercontent.com/8422414/165361814-0beb8583-4aea-4032-b12b-c01d4ccbe638.png).

However, using `texCoord_0` you get coordinates that do follow the surface of the model, even when the model is animated.
![image](https://user-images.githubusercontent.com/8422414/165360767-2347cbbb-a497-4aec-8485-427d87423430.png)

Options:

1. Leave the CesiumJS implementation alone, let the user provide texture coordinates (and or `EXT_mesh_features`) if procedural texturing is desired. This avoids complexity in the implementation. Though it might be helpful to document best practices.
2. Add more struct fields for the vertex positions before applying morphing/skinning. This adds some complexity and more struct fields in the shader, but it may be easier to use in the shader.

Personally I'm leaning towards option 1) for a couple reasons:

1. Geospatial data is very rarely animated in the glTF sense. This functionality might not be needed very often.
2. Even if you had something like `fsInput.attributes.unmorphedPositonMC`, the usefulness highly depends on the data and the desired shader behavior. Might not be worth the complexity of adding more struct fields.

Contributor guide

Open the contributing guide

Research direction

Start with the linked Sandcastle example and compare CustomShader's positionMC behavior with texCoord_0 during CesiumMan animation. Review #10314 and the proposed options for pre-morph or pre-skin positions versus author-provided coordinates; done requires a decided API direction and documented implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
computer-graphics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.