How to ControlNet with semantic (non-pixel-aligned) concepts?

Nobody has claimed this yet.

**I have a different ControlNet case, and I'd love feedback on how to get it working**

I am trying to train a ControlNet from SignWriting (lexical writing of sign language) and illustrations. (https://github.com/sign-language-processing/signwriting-illustration) Unlike other ControlNet examples where the control is pixel-aligned, here, my image represents semantically what needs to be generated, similar to the text.

Every video in my corpus is annotated with SignWriting, and an illustration.
All of the information about the sign is indeed represented in SignWriting.

I create the prompts using `gpt-4-vision` to include additional information, but no information about the sign (for example, gender, but not the direction of the hands)

How my dataset looks like:

All images are then created at 512x512, for example:
`An illustration of a woman with short hair, with orange arrows. The background is white and there is a watermark text '@signecriture.org `
| control | illustration |
| ----- | ----- |
|  |  |

### Results

Using the above example `control` with the prompt `An illustration of a man with short hair, with orange arrows. The background is white.`, I tried generating 5 different illustrations. My expectation is that the appearance of the person might change somewhat, but that the hand positions and arrow directions should be consistent with the SignWriting representation.

Training with `sd_locked=True, only_mid_control=False` yields a system that can not adequately illustrate:

I played with ddim_steps, scale and strength but all are consistently bad


Training with `sd_locked=False, only_mid_control=False`, the system can now illustrate, but the results are not consistent:

Playing with ddim_steps, scale and strength does not change the results much


Is there anything I can do to get it working?

Start by reviewing the ControlNet training and generation setup described in the issue, especially the sd_locked and only_mid_control settings and the ddim_steps, scale, and strength experiments. Investigate how semantic SignWriting conditioning could preserve hand positions and arrow directions across generated illustrations; done means the expected semantic structure remains consistent while appearance can vary.

Written by the indexing model from the issue text.