ByteDance-Seed / ByteDance-Seed/Depth-Anything-3
Can't train 3DGS on export_to_colmap results
- Dominant language
- Python
- Stars
- 6.3k
- Forks
- 702
- PR merge metrics
- No merged PRs in 30d
Description
I exported predictions to Colmap format using `export_to_colmap` function: it generates `points3D.bin`, `images.bin`, `frames.bin`, `cameras.bin` and `rigs.bin`. I note here that for 300 images it generates 20M points with a confidence percentile of 70.
Next, I tried to run `train.py` from original gaussian splatting repo (`graphdeco-inria`) but I consistently get this error:
```
Converting point3d.bin to .ply, will happen only the first time you open the scene. [28/11 23:25:12]
Loading Training Cameras [28/11 23:26:20]
Loading Test Cameras [28/11 23:26:48]
Number of points at initialisation : 12827810 [28/11 23:26:49]
[gpu-health] [WARN] GPU-a2edc31e-63f6-ebc5-f156-ed40ee9f221b: XID: NVRM: Xid (PCI:0000:ca:00): 31, pid=1907695, name=exe, Ch 00000008, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC11 GPCCLIENT_T1_0 faulted @ 0x2b0d_7b7bf000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
Training progress: 0%| | 0/7000 [00:00
training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from)
File "/app/gaussian-splatting/train.py", line 142, in training
loss.backward()
File "/opt/conda/envs/torch/lib/python3.10/site-packages/torch/_tensor.py", line 521, in backward
torch.autograd.backward(
File "/opt/conda/envs/torch/lib/python3.10/site-packages/torch/autograd/__init__.py", line 289, in backward
_engine_run_backward(
File "/opt/conda/envs/torch/lib/python3.10/site-packages/torch/autograd/graph.py", line 769, in _engine_run_backward
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Training progress: 0%| | 0/7000 [00:00
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the export_to_colmap output files—points3D.bin, images.bin, frames.bin, cameras.bin, and rigs.bin—and reproduce the failure with graphdeco-inria's train.py. Inspect the point-count initialization and the backward-pass CUDA error. Done means documenting whether the generated point count or COLMAP output is unsupported and identifying the required correction or limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100