CesiumGS / CesiumGS/cesium

Clipping for Gaussian splats

Open
#13,474 0 comments 0 reactions 0 assignees View on GitHub
category - clipping category - gaussian splatting type - enhancement
Dominant language
JavaScript
Stars
15.7k
Forks
3.9k
Avg merge
4d 6h
Merged PRs (30d)
34

Description

### Feature

Gaussian splat data sets are usually generated by reconstructing the splats from a set of photos. The result inevitably includes points/splats that may not be part of the actual object of interest. For example, when capturing a building from a drone scan, then the data set will contain splats that have been reconstructed from the surrounding environment or from the sky:

Image

There are different options for where, when, and how different forms of "clipping" could happen. The main categories are whether it happens _before_ or _after_ the tiling process.

### Option 1: Preprocessing

This is, strictly speaking, not an option that can be implemented sensibly in CesiumJS itself. But if a Gaussian splat tileset is not generated with a photo reconstruction, but from a PLY file, then it is possible to use online editors like https://superspl.at/editor to edit the PLY file manually, before tiling:

Image

### Option 2: Postprocessing

For existing Gaussian splat tilesets (regardless of whether they have been created with a reconstruction process, or from a PLY file), there is no appropriate tooling for clipping. This clipping would have to happen on the level of the whole tileset. Specifically: The clipping would have to take the LOD hierarchy into account, clip the splat data sets for every LOD, and write out the result. (Or... exactly _not_ write out whatever has been completely clipped away).

(I have some local tooling for stuff like this, and a basic form of that could easily - nearly trivially - be implemented as part of the 3D Tiles Tools. But depending on the exact requirements (e.g. the actual clipping shape, and _how it is defined_), a more interactive solution would be desirable).

### Option 3: Runtime clipping

Regardless of whether tiling happens based on photo reconstruction or based on PLY data, and regardless of where any clipping of the data already happened, it should _always_ be possible to apply the standard clipping mechanisms of CesiumJS to existing tilesets. This includes clipping planes and clipping polygons, and preferably additional, more powerful clipping functionalities (namely, clipping _volumes_).

There are some structures that could either be used for implementing this functionality, or that could be seen as "workarounds" for the current state. The following is an example sandcastle **that will not show the desired effect on the sandcastle that is currently depoloyed**, but it **will** work once https://github.com/CesiumGS/cesium/pull/13418 is merged: It uses a custom shader to cut away parts of a splat data set:

https://sandcastle.cesium.com/index.html#c=jVZrb9s2FP0rnL9EBhwpnpe2SJ1grddsBvJC7KQYICClJMoiQpEaSdl1ivz3Xb4kxwuyAonNx33fcy6dJOiW/NNSSRTKW6VFjVSFCyJRKSRSDcNa/ZLyJEELQlCldaNOkmRFddVmcS7qZEYUbes/F0luF0nTMpaMJ7+NP6Q85bngSqM1JRuweIo42SCnEN/bsygdOL2Z4BpTTmQ6GH40muBxJgnWBOkK/ikjiugRwkWBqEZa2GOVE07gkBfoSUDocEx18Op1wC3eYNDxjt3X5I+lu45LKeq54J8UbOZFNHn37v37DxMThAs7tk7iRtKaaromKoYYIm98R8wEsBRRylEXrVnvpPwXwQXlqxuq8+oW8xWxwihcX2JdxVrcghDmKjqKj4ajtwQOx8e9yK9HIG/X5mS/hDg0VVdYo4KqHMtCIcwYKiVe1YRrhTYV4aaqVBoJjXlOrA1fa6iCBguiNNaaikiCqEIr68Aa5nCxggJxJCHCVsVWewmqLaeAphocSrJry/TNbJ28MW2bao332l54QyHaDK6JbX/mLGWi5aaoaC1YW+9E6ULvoRNbW95TsEUKqkkBW1051xBRjpVmBN3NA45yRpuFK98LBM8sXdxN9MOUPuR5gn64rrQPLp7uALCxbchJMHHnFJZwFt9/mU1GQWqNWQtiu96w1LDCfGKQMULhIwDgeRRcuhx/xuX5xfWn5b7Pq7bOAM4314v5cn7/5WF+dT6/mi//fuHHfTK6qjQU/1IUhHUeLnZP47uri7nzEZDmKrYk3/UJ+gb0EbTo7i5hBkTnfjPnTatRqez3CFEuYJs/1Q+1sXxpcEcxQ7VfDEPGa5JPAMJlCf3y2jHWWtKshRLGjVBAZMG/ztBh16GPTrVkwhHEwh/0GeErXUXG2tDL0BJFncRZV/BhX3BPMC//nHL4+wY1eO54CdOGrrjDnGem2Bt0wA3KIVDM6JPFulXsqGQxW5ASt0y71qmUB6znO9CEJHoEg/9+E4OohwPM4VCJdDDqOBPYtXCUdALDN424Yhgjx0dduoZ6Zkwi2zk3hWB4AL+0FIz9LwWNroUTJGOrHDB+7Kbec3B01xThyXj5mh2EyA5CCXuXtnpW3bwF7qgP1o5FeD7yyoxsKHLZ8tzgB7XWmY3rHBTNixZ5FLiw/bQ59ZyKujRiDxmLkJ+pZS//7DP9Ck82RPDfmu3k1SWRcs/NRy7yR6BRrCXOH/uAbE/dqykEy7BBTSHy1tAwXhH9hRGz/LyFFzIdeBn3VO9bxk3Dtp+pxY3qPYyCZaNjfltEzh/HMLUp71vsK2hY1terwup6w2+kaIjU28goDTvC7UUQWGjivs4UkWucMbIbiVXvxFSbqRyGA4leaahrkaPwYDSYKr1l5Myc/U7rRkiNWsmiOE40qc2PJaKSrM0fDQeV7dc0CSrTgq4RLU5f+ckDGMBKwU0Jv54WwPd0cDZNQP6FGswmU9TrNZEMb0HEhDGtxmcX7iKO42kCW+N0X7drmVOidrSahwHupAE2BFFTDjtglFnj77AeH7ud0qSxV2PYQI3wYQYNhhP/Zjh4jhyTHAdP0IF1cgAayStONcx/YxmShd3xW3a9BZfUvw

It defines a custom shader that simply discards all fragments when their distance to the center of the bounding sphere of the tileset is larger than a user-selected threshold:

Image

This could be extended arbitrarily (i.e. to define clipping planes or volumes), but can be seen as a "proof of concept" for now.

### AI acknowledgment

- [ ] I used AI to generate this feature request.
- [ ] (If the above is checked) I have reviewed the AI-generated content before submitting.

Contributor guide

Open the contributing guide

Research direction

Start with the runtime clipping section, the linked Sandcastle example, and CesiumJS pull request #13418. Clarify whether the work targets runtime clipping, postprocessing, or both, including the clipping shapes and LOD behavior. Done should mean a defined clipping capability for Gaussian splat tilesets, with an example demonstrating the intended result.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
computer-graphics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.