NVIDIA / NVIDIA/harmonizer

[BUG] Reproduction gap between Hugging Face checkpoint model and paper demo results

Open
#5 3 comments 0 reactions 1 assignee View on GitHub

@daehyoungko is already working on this.

Since Jul 21, 2026.

bug
Dominant language
Python
Stars
68
Forks
8
Avg merge
2h 25m
Merged PRs (30d)
1

Description

Describe the bug

The official checkpoint (diffusion_harmonizer.pkl from Hugging Face)
does not reproduce the qualitative results shown in the paper/demo.
Specifically, the color/tone harmonization between inserted foreground
objects and the background scene is nearly absent, and shadow synthesis
is significantly weaker than demonstrated.

Using the first demo example from the paper(the scene with the Lightning McQueen-like
cartoon car inserted into a real street intersection):

  • Expected (paper/demo): The foreground cartoon car blends
    naturally into the scene — color tone matches the environment lighting,
    and a realistic shadow is synthesized beneath/around the car.
  • Actual (official checkpoint): The foreground McQueen car shows
    almost no color/tone harmonization with the background. The shadow
    effect around the foreground car is unsatisfactory. The two background
    cars look relatively reasonable, but the prominent foreground car does
    not blend into the scene as showcased.

Steps/Code to reproduce bug

  1. Download the official checkpoint diffusion_harmonizer.pkl from
    Hugging Face.
  2. Run inference using the provided inference script with the demo
    input (Lightning McQueen scene from 1st page on paper):
python inference_pix2pix_turbo_harmonizer.py \
--input_image examples/ \
--model_path diffusion_harmonizer.pkl \
--timestep 250 \
--nontemporal
  1. Compare the output against the paper/demo qualitative results.
  2. I also tried different inference timesteps (e.g., 100, 200, 250, 500)
    — the overall result still does not match the demo quality.

Expected behavior

The output should match the qualitative results shown in the paper and
demo materials:

  • Foreground objects should have their color/tone harmonized with the
    background scene.
  • Realistic shadows should be synthesized for inserted objects.
  • The overall result should look natural, as demonstrated in the paper
    figures.

Environment overview (please complete the following information)

  • Environment location: Docker
  • Method of Deployment: Docker
  • Built harmonizer environment on top of the base image per the
    provided Dockerfile.

Environment details

  • Hardware: NVIDIA RTX 4090
  • Operating System: Ubuntu 22.04 (Linux 5.15.0)
  • Number of physical nodes: 1 (inference only)
  • Network connectivity between the nodes: N/A (single-node inference)

Additional context

I would like to ask:

  1. Was the paper/demo result generated using the same checkpoint
    (diffusion_harmonizer.pkl) that is currently released on Hugging Face?
  2. Were there any additional training datasets, fine-tuning stages, or
    internal checkpoints used for generating the demo examples that differ
    from the released checkpoint?
  3. The official GitHub repository does not appear to implement the
    multi-scale VGG perceptual loss described in the paper (Section 7.2). Is
    the released checkpoint trained with the loss recipe described in
    the paper?

Thank you for this excellent work — any clarification would be greatly
appreciated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.