lllyasviel / lllyasviel/ControlNet

Support for 16-Bit Color Depth

Open
#414 4 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
34.1k
Forks
3k
PR merge metrics
No merged PRs in 30d

Description

The depth models for ControlNet can produce some really high quality images that have a phenomenal understanding of relative perspective, especially if used with a dedicated depth map as opposed to a preprocessor.

However, it has one very serious drawback:

From my testing, it seems as though ControlNet only supports 8-bit color depth. What this means is that there are only 255 levels of depth that ControlNet can interpret, and on images where there is a big difference in positioning between scene elements a lot of fine details can be lost. This can be an issue even in relatively small scenes where there is only a maximum depth of around 15 meters between scene elements, but it can especially become a massive issue on larger scenes like landscape shots where there can sometimes be multiple kilometers between elements.

This issue can be somewhat remedied by combining both a depth and normal map, but allowing for 16-bits of color depth, or 65,536 levels of depth, if a user chooses to toggle this option would allow for far more precision in generated images at the cost of performance, which I believe would be a trade-off a lot of people would be willing to take if they could.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue does not name any files, tests, or entry points, so begin by locating the depth-model input and preprocessing paths in the repository. The work is done when an optional 16-bit depth path is implemented, preserves the requested precision, and has tests or examples covering the new option and its trade-offs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.