kwea123 / kwea123/nerf_pl

How to get real depth values from approximated depth in certain ray.

Open
#124 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
458
PR merge metrics
No merged PRs in 30d

Description

Hi, I'm appreciate to your great work!

I'm implementing some light-field based neural network which have different input structure for MLP.

The MLP outputs the rgb color and depth values which are in range [0, 1] in NDC coordinates by sigmoid function to corresponding ray.

However, my questions is follows

1. For NDC coordinates, if I correctly understand, the scene bounded in `near(0)` and `far(1)` plane. Then, do all components exist between near and far plane?

2. As we can see the `get_rays` functions, **z** value is instantiate with `-1`. I can sure that camera origin is world coordinates from the evidence that `c2w[:3,-1]` part in following code. As I understand, the rays are still in world coordinate since the magnitude of direction vector doesn't matter. Am I right?
image

3. I get some depth value and compute the corresponding points using follow code. I intend the code works regardless of the coordinate system, NDC or world. z_vals contains `N_rays x [near, far]` values. Is this code right..?
image

4. If points are correctly computed in given ray with some depth values, how to compute the real depth value when NDC coordinate system is used?
For example, if I have P(Xp, Yp, Zp) in NDC coordinates, following code can back project to real depth value?
I use same coordinate system to NeRF, (right, up, backward), in world and I guess NDC coordinate system is (right, up, forward).

I read about the [issue](https://github.com/bmild/nerf/issues/35) you raised in NeRF original GitHub, but I think it still not work properly for my code.

For given `ndc_pts`, which have `N_rays x (Xp, Yp, Zp)`, we back-project `ndc_pts` to world following your issue.
image

But what I found in your GitHub [issue](https://github.com/kwea123/nerf_pl/issues/48), which refer the necessary of back project with `K^-1 * d * (x y 1)`.
image

For depth direction, I think `depth_real = 1/(1-depth_ndc)` is different because of the different coordinate system.
Then, does `K^-1 * d * (x y 1)` mean same process with the mentioned code from NeRF author?(which I used).

Thanks to read my questions.
If you reply to this issue, it will be really helpful...!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the get_rays function and the NDC coordinate calculations shown in the issue, then compare them with the linked NeRF and nerf_pl discussions. Resolve how ray points and depth should be transformed between NDC and world coordinates, and document the correct real-depth calculation for the stated coordinate conventions.

Written by the indexing model from the issue text.

Assessment

Tech stack
pytorch
Domain
computer-vision, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.