ByteDance-Seed / ByteDance-Seed/Depth-Anything-3

Does DA3-Streaming support pose-conditioned depth estimation with known camera poses?

Open
#264 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.3k
Forks
702
PR merge metrics
No merged PRs in 30d

Description

**Hi, thanks for the great work on Depth Anything 3 and the new DA3-Streaming mode.**

I have a question about the current and future support for **pose-conditioned depth estimation** in **DA3-Streaming**.

From the main DA3 API, it seems that pose-conditioned inference is supported by passing known `extrinsics` and `intrinsics` to `model.inference(...)`, and the model can output per-frame depth and confidence maps in the input camera coordinate system.

However, in the current DA3-Streaming wrapper, the usage seems to be mainly:

```bash
python da3_streaming.py --image_dir ./path_of_images
```

and internally it appears to call something like:

```python
predictions = self.model.inference(images, ref_view_strategy=...)
```

without passing user-provided `extrinsics` / `intrinsics`.

**So I would like to ask:**

1. **Does DA3-Streaming currently support pose-conditioned depth estimation with known camera poses?**
- For example, can we provide a sequence of RGB images together with fixed `extrinsics` and `intrinsics`, and get streaming depth/confidence predictions aligned to those input poses?

2. If it is not currently supported, is there any plan to support this in the future?

3. If the core DA3 model already supports this, would it be reasonable to modify the streaming pipeline so that each chunk calls:

```python
model.inference(
images,
extrinsics=known_extrinsics,
intrinsics=known_intrinsics,
align_to_input_ext_scale=True,
)
```

while avoiding the streaming pipeline's own predicted-pose alignment?

Our use case is long multi-view / video depth estimation where camera poses are already known from an external system. We want to use DA3-Streaming mainly for scalable depth and confidence prediction, while keeping the output geometry fixed in the provided camera coordinate system.

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading da3_streaming.py and the main DA3 model.inference API, focusing on how image chunks, extrinsics, intrinsics, and pose alignment are handled. Verify whether known poses can be passed through streaming without predicted-pose alignment; done means a supported behavior or a documented limitation, with validation of depth and confidence outputs in the provided camera coordinate system.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.