CUDA OOM during render_utils.render_video (diff-gaussian-rasterization) on A100 40 GB inside Singularity

Running the sample pipeline in a Singularity container on an NVIDIA A100-40 GB causes a CUDA out-of-memory error inside diff_gaussian_rasterization when the first call to render_utils.render_video() is made.

Environment

GPU            : NVIDIA A100-SXM4-40GB
Driver / CUDA  : 555.42.02 / 12.4
PyTorch        : 2.3.0 + cu124
diff-gaussian-rasterization commit : 59f5f77 (2025-07-31)
Trellis        : gqk/TRELLIS-image-large-fork (pulled 2025-08-04)
Container      : Singularity 4.1.3 (Ubuntu 22.04 base)
Python         : 3.10.14

Exact command

#!/bin/bash
#SBATCH --gres=gpu:a100_3g.40gb:1

apptainer exec --nv env/trellis.sif python3 memory_check.py

Error log

Traceback (most recent call last):
  File "/home/nyamaguchi/0_ws/TRELLIS/example.py", line 39, in <module>
    video = render_utils.render_video(outputs['gaussian'][0])['color']
  File "/home/nyamaguchi/0_ws/TRELLIS/trellis/utils/render_utils.py", line 100, in render_video
    return render_frames(sample, extrinsics, intrinsics, {'resolution': resolution, 'bg_color': bg_color}, **kwargs)
  File "/home/nyamaguchi/0_ws/TRELLIS/trellis/utils/render_utils.py", line 81, in render_frames
    res = renderer.render(sample, extr, intr, colors_overwrite=colors_overwrite)
  File "/home/nyamaguchi/0_ws/TRELLIS/trellis/renderers/gaussian_render.py", line 223, in render
    render_ret = render(camera_dict, gausssian, self.pipe, self.bg_color, override_color=colors_overwrite, scaling_modifier=self.pipe.scale_modifier)
  File "/home/nyamaguchi/0_ws/TRELLIS/trellis/renderers/gaussian_render.py", line 124, in render
    rendered_image, radii = rasterizer(
  File "/opt/conda/envs/trellis/lib/apython3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/trellis/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/tmp/extensions/mip-splatting/submodules/diff-gaussian-rasterization/diff_gaussian_rasterization/__init__.py", line 216, in forward
    return rasterize_gaussians(
  File "/tmp/extensions/mip-splatting/submodules/diff-gaussian-rasterization/diff_gaussian_rasterization/__init__.py", line 32, in rasterize_gaussians
    return _RasterizeGaussians.apply(
  File "/opt/conda/envs/trellis/lib/python3.10/site-packages/torch/autograd/function.py", line 574, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/tmp/extensions/mip-splatting/submodules/diff-gaussian-rasterization/diff_gaussian_rasterization/__init__.py", line 94, in forward
    num_rendered, color, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 66847744.04 GiB. GPU 0 has a total capacity of 39.25 GiB of which 33.42 GiB is free. Process 2877885 has 3.04 GiB memory in use. Process 3024965 has 2.83 GiB memory in use. Including non-PyTorch memory, this process has 5.78 GiB memory in use. Of the allocated memory 5.47 GiB is allocated by PyTorch, and 55.73 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

example.py

import os
# os.environ['ATTN_BACKEND'] = 'xformers'   # Can be 'flash-attn' or 'xformers', default is 'flash-attn'
os.environ['SPCONV_ALGO'] = 'native'        # Can be 'native' or 'auto', default is 'auto'.
                                            # 'auto' is faster but will do benchmarking at the beginning.
                                            # Recommended to set to 'native' if run only once.

import imageio
from PIL import Image
from trellis.pipelines import TrellisImageTo3DPipeline
from trellis.utils import render_utils, postprocessing_utils

# Load a pipeline from a model folder or a Hugging Face model hub.
pipeline = TrellisImageTo3DPipeline.from_pretrained("gqk/TRELLIS-image-large-fork")
pipeline.cuda()

# Load an image
image = Image.open("assets/example_image/T.png")

# Run the pipeline
outputs = pipeline.run(
    image,
    seed=1,
    # Optional parameters
    # sparse_structure_sampler_params={
    #     "steps": 12,
    #     "cfg_strength": 7.5,
    # },
    # slat_sampler_params={
    #     "steps": 12,
    #     "cfg_strength": 3,
    # },
)
# outputs is a dictionary containing generated 3D assets in different formats:
# - outputs['gaussian']: a list of 3D Gaussians
# - outputs['radiance_field']: a list of radiance fields
# - outputs['mesh']: a list of meshes

# Render the outputs
video = render_utils.render_video(outputs['gaussian'][0])['color']
imageio.mimsave("sample_gs.mp4", video, fps=30)
video = render_utils.render_video(outputs['radiance_field'][0])['color']
imageio.mimsave("sample_rf.mp4", video, fps=30)
video = render_utils.render_video(outputs['mesh'][0])['normal']
imageio.mimsave("sample_mesh.mp4", video, fps=30)

# GLB files can be extracted from the outputs
glb = postprocessing_utils.to_glb(
    outputs['gaussian'][0],
    outputs['mesh'][0],
    # Optional parameters
    simplify=0.95,          # Ratio of triangles to remove in the simplification process
    texture_size=1024,      # Size of the texture used for the GLB
)
glb.export("sample.glb")

# Save Gaussians as PLY files
outputs['gaussian'][0].save_ply("sample.ply")

Start with example.py and reproduce the failure using the exact Apptainer command and environment details. Trace render_utils.render_video through trellis/renderers/gaussian_render.py into the diff-gaussian-rasterization call shown in the traceback. Done means the sample Gaussian render completes on the stated A100 without the CUDA out-of-memory failure.