facebookresearch / facebookresearch/hyperreel
Any tips for training on my own dataset / fixing scale from COLMAP?
- Dominant language
- Python
- Stars
- 481
- Forks
- 33
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
First of all, thanks for releasing the code for this impressive method!
I have been trying to train this on my own dataset (it's front-facing).
After a bit of experimentation, I can get ok-ish results. To do this, I extracted the poses from COLMAP then used `imgs2poses.py` in the LLFF repo to obtain the `poses_bounds.npy` for the 16 cameras in my dataset. Then, I created a `.mp4` video for each camera to get the dataset into an identical format as per `neural_3d` in your repo.
From there, the only change I needed to make to get training to work was to expand the `grid_bounds` in `neural_3d_z_plane.yaml` from:
```
# Grid bounds
aabb: [[-1.0, -1.0, -1.0], [1.0, 1.0, 1.0]]
```
to:
```
# Grid bounds
aabb: [[-5.0, -5.0, -5.0], [5.0, 5.0, 5.0]]
```
Then I get OK results.
So it seems like COLMAP estimated my scene to have a somewhat larger scale than in the neural_3d scenes.
I am wondering if based on this, you had any ideas as to how I can improve performance to be closer to the level of neural_3d? I just have the feeling the code is optimised for a (-1,1) aabb box. But digging into the code, COLMAP, etc., it's not super obvious how I can rescale my scene, or what I should change in hyperreel to make it better suit my scene.
Just thought I would share this result, and yeah if you have any tips/ideas I'd really appreciate it! I'll also share any progress I manage to make on my own.
Contributor guide
Research direction
Start by comparing the 16-camera poses_bounds.npy produced by imgs2poses.py with the neural_3d dataset and inspect the aabb in neural_3d_z_plane.yaml. Measure whether rescaling the COLMAP scene or changing the [-1,1] bounds affects training quality; done means documenting a reproducible configuration that improves the custom dataset toward neural_3d results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100