Training a ControlNet to generate furnished room -> empty room (and vice versa). Improvement plateau...

Nobody has claimed this yet.

I'm working on a project to take images of furnished rooms and remove all the furniture. I've got a large dataset of image pairs. I'm not using any preprocessing on the images so as to allow the model to preserve details of the original image (wall color, floor material, etc.).

After training on a 4090 for about 5 days, and I'm no longer seeing any improvement (see examples below).

I'm looking to get tips about where to go from here.

- Does it just need to be trained longer?
- Do I need to adjust the learning rate?
- Should I spend more time cleaning the dataset (a small % of the dataset is probably bad, as you can see in one of the examples below, the target image is dark).
- Should I preprocess the image to simplify this? (i.e MLSD) It would lose the details of the original, but maybe at least will provide better output for final image.
- Perhaps ControlNet isn't the right arch for this and instead use pix2pix?

Thanks for the help!

## Example 1

Source:

Target:

Model Result:

## Example 2

Source:

Target:

Model Result:

## First Training Run

## Second Training Run

The issue names no repository files or tests; start by reviewing the paired image dataset, the first and second training runs, and the listed ControlNet-versus-pix2pix questions. A useful outcome would be a supported direction for addressing the plateau, including whether to change training, clean or preprocess data, or use another architecture.

Written by the indexing model from the issue text.