isl-org / isl-org/Open3D

Uninitialized UV0 corrupts legacy TriangleMesh rendering on Mesa Lavapipe

Open
#7,565 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
14k
Forks
2.6k
Avg merge
5d 18h
Merged PRs (30d)
6

Description

## Description

With the official Open3D 0.20.0 Python wheel on Ubuntu 26.04, `OffscreenRenderer` produces corrupted output when it uses Mesa Lavapipe software Vulkan. A small six-sided cylinder contains large white polygons that do not belong to the material or lighting.

The reproducer creates a cylinder, reconstructs the same mesh through the public `TriangleMesh(vertices, triangles)` constructor, recomputes its normals, and renders it. Passing `--control` renders the original factory-created mesh instead; that output is clean.

This does not require ambient occlusion, shadows, a ground plane, or application-specific geometry.

## Steps to reproduce

Install the official wheel and NumPy, save the script below as `repro.py`, then force Mesa Lavapipe:

```bash
python -m venv .venv
.venv/bin/pip install open3d==0.20.0 numpy

EGL_PLATFORM=surfaceless \
VK_DRIVER_FILES=/usr/share/vulkan/icd.d/lvp_icd.json \
.venv/bin/python repro.py
```

Run the clean control with:

```bash
EGL_PLATFORM=surfaceless \
VK_DRIVER_FILES=/usr/share/vulkan/icd.d/lvp_icd.json \
.venv/bin/python repro.py --control
```

```python
"""Open3D 0.20 software-Vulkan rendering corruption reproducer."""

import argparse

import numpy as np
import open3d as o3d
from open3d.visualization import rendering

parser = argparse.ArgumentParser()
parser.add_argument("--control", action="store_true")
args = parser.parse_args()

# This order is required for the reduced, allocation-sensitive reproduction.
renderer = rendering.OffscreenRenderer(1920, 1080)
renderer.scene.set_background([0.58, 0.58, 0.58, 1.0])
renderer.scene.show_skybox(False)
renderer.scene.view.set_post_processing(True)
renderer.scene.view.set_antialiasing(True)
renderer.scene.view.set_ambient_occlusion(False)
renderer.scene.view.set_shadowing(False, rendering.View.ShadowType.VSM)

mesh = o3d.geometry.TriangleMesh.create_cylinder(
radius=0.5, height=1.5, resolution=6, split=6
)
mesh.translate([0.0, 0.0, 0.75])
mesh.compute_vertex_normals()

# Rebuild the identical mesh through the public array constructor.
# --control skips this block and renders cleanly.
if not args.control:
mesh = o3d.geometry.TriangleMesh(
o3d.utility.Vector3dVector(np.asarray(mesh.vertices).copy()),
o3d.utility.Vector3iVector(np.asarray(mesh.triangles).copy()),
)
mesh.compute_vertex_normals()

material = rendering.MaterialRecord()
material.shader = "defaultLit"
material.base_color = [0.263, 0.117, 0.552, 1.0]
material.base_roughness = 0.55
renderer.scene.add_geometry("mesh", mesh, material)

renderer.scene.scene.set_sun_light(
[-0.4, -0.6, -1.0], [1.0, 1.0, 1.0], 75_000
)
renderer.scene.scene.enable_sun_light(True)
renderer.setup_camera(
45.0, [0.0, 0.0, 0.4], [2.3, -2.3, 1.8], [0.0, 0.0, 1.0]
)

pixels = np.asarray(renderer.render_to_image())
output = "control.png" if args.control else "corrupt.png"
o3d.io.write_image(output, o3d.geometry.Image(pixels), 9)
bright_pixels = int(np.all(pixels[:, :, :3] > 200, axis=2).sum())
print(f"Open3D {o3d.__version__}; wrote {output}; bright pixels: {bright_pixels}")
```

## Actual behavior

`corrupt.png` contains large, flat white polygons across the purple cylinder.

The relevant output is:

```text
[Open3D INFO] EngineInstance: Vulkan software device detected; using Filament's Vulkan backend
Open3D 0.20.0; wrote corrupt.png; bright pixels: 47600
FEngine resolved backend: Vulkan
Vulkan device driver: llvmpipe Mesa 26.0.8-1ubuntu0.3 (LLVM 21.1.8)
Selected physical device 'llvmpipe (LLVM 21.1.8, 256 bits)' from 1 physical devices.
```

I ran the reproducer three times on the same host. All three corrupted images were byte-for-byte identical:

```text
SHA256 ef247ff1ac757b9f5a6894723a8447eeba84ff50a01b9aec75b5da8334760d00
```

The control reports 50 bright pixels and does not contain the white polygons.

## Expected behavior

Reconstructing a `TriangleMesh` from copies of its vertex and triangle arrays and recomputing normals should render the same geometry without white polygons.

## Environment

- Open3D: 0.20.0 official Python wheel
- Python: 3.12.13
- OS: Ubuntu 26.04.1 LTS
- Kernel: 7.0.0-31-generic x86_64
- CPU: Intel Core Ultra 9 285K
- `mesa-vulkan-drivers`: 26.0.8-1ubuntu0.3
- `libvulkan1`: 1.4.341.0-1
- Vulkan device: llvmpipe, Mesa Lavapipe software Vulkan
- LLVM: 21.1.8
- Headless rendering; `EGL_PLATFORM=surfaceless`

## Likely cause in `TriangleMeshBuffers.cpp`

The legacy `TriangleMesh` path appears to upload an uninitialized UV attribute:

1. [`CreateColoredBuffers`](https://github.com/isl-org/Open3D/blob/main/cpp/open3d/visualization/rendering/filament/TriangleMeshBuffers.cpp#L189-L226) allocates `TexturedVertex` storage with `malloc`.
2. It writes position, tangent, and color for each vertex, but never writes `TexturedVertex::uv`. The struct's default member initializer does not run for storage obtained through `malloc`.
3. In [`ConstructBuffers`](https://github.com/isl-org/Open3D/blob/main/cpp/open3d/visualization/rendering/filament/TriangleMeshBuffers.cpp#L443-L473), the no-UV mesh takes the `CreateColoredBuffers` branch and then sets `has_uvs = true`.
4. [`BuildFilamentVertexBuffer`](https://github.com/isl-org/Open3D/blob/main/cpp/open3d/visualization/rendering/filament/TriangleMeshBuffers.cpp#L105-L139) consequently exposes `UV0` to Filament, pointing it at the unwritten bytes.

Two diagnostics support this explanation:

- Adding initialized zero UVs before `add_geometry` makes the output clean and byte-for-byte identical to the control:

```python
mesh.triangle_uvs = o3d.utility.Vector2dVector(
np.zeros((3 * len(mesh.triangles), 2))
)
```

- Changing glibc's heap fill through `MALLOC_PERTURB_` changes or removes the corruption. With the normal environment the render has 47,600 bright pixels; `MALLOC_PERTURB_=1` changes it to 15,356; values 42, 85, 170, and 254 produce the clean control image exactly.

A targeted fix may be to initialize `element.uv` in `CreateColoredBuffers`, for example:

```cpp
TexturedVertex& element = vertices[i];
element.uv = kDefault.uv;
```

This matches the fallback used by the legacy point-cloud buffer builder and the tensor triangle-mesh path. `has_uvs = true` should remain because Open3D's built-in lit and unlit materials require `UV0`.

## Additional observations

- Ambient occlusion is explicitly disabled in the reproducer.
- No ground plane or application-specific code is involved.
- Creating `OffscreenRenderer` before rebuilding the mesh is required for this reduced case. Rebuilding the mesh before creating the renderer rendered cleanly in my test.
- The symptom is topology and allocation sensitive. Some nearby cylinder resolutions render cleanly, while resolutions 6, 11, 12, 15, 18, 22, 24, 64, and 100 showed corruption on this host.
- The original, larger scene showed horizontal white bands. This reduction turns the symptom into white polygons on individual faces.
- This was found while testing the software-Vulkan path introduced by https://github.com/isl-org/Open3D/pull/7550.

I have prepared a focused Open3D patch that initializes the fallback UV and a software-Vulkan regression test using this conventional `TriangleMesh` case at 1920x1080. I can link the pull request here once submitted.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with CreateColoredBuffers and ConstructBuffers in cpp/open3d/visualization/rendering/filament/TriangleMeshBuffers.cpp, then trace how BuildFilamentVertexBuffer exposes UV0. Run repro.py with the Lavapipe environment and compare corrupt.png with control.png; done means the reconstructed mesh renders cleanly and the reported software-Vulkan regression test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
computer-graphics, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.