facebookresearch / facebookresearch/hyperreel

Any tips for training on my own dataset / fixing scale from COLMAP?

Open
#11 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
481
Forks
33
PR merge metrics
No merged PRs in 30d

Description

Hi there,

First of all, thanks for releasing the code for this impressive method!

I have been trying to train this on my own dataset (it's front-facing).

After a bit of experimentation, I can get ok-ish results. To do this, I extracted the poses from COLMAP then used `imgs2poses.py` in the LLFF repo to obtain the `poses_bounds.npy` for the 16 cameras in my dataset. Then, I created a `.mp4` video for each camera to get the dataset into an identical format as per `neural_3d` in your repo.

From there, the only change I needed to make to get training to work was to expand the `grid_bounds` in `neural_3d_z_plane.yaml` from:

```
# Grid bounds
aabb: [[-1.0, -1.0, -1.0], [1.0, 1.0, 1.0]]
```

to:

```
# Grid bounds
aabb: [[-5.0, -5.0, -5.0], [5.0, 5.0, 5.0]]
```

Then I get OK results.

So it seems like COLMAP estimated my scene to have a somewhat larger scale than in the neural_3d scenes.

I am wondering if based on this, you had any ideas as to how I can improve performance to be closer to the level of neural_3d? I just have the feeling the code is optimised for a (-1,1) aabb box. But digging into the code, COLMAP, etc., it's not super obvious how I can rescale my scene, or what I should change in hyperreel to make it better suit my scene.

Just thought I would share this result, and yeah if you have any tips/ideas I'd really appreciate it! I'll also share any progress I manage to make on my own.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the 16-camera poses_bounds.npy produced by imgs2poses.py with the neural_3d dataset and inspect the aabb in neural_3d_z_plane.yaml. Measure whether rescaling the COLMAP scene or changing the [-1,1] bounds affects training quality; done means documenting a reproducible configuration that improves the custom dataset toward neural_3d results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.