Unsatisfactory human pose training result

Thanks for your great contribution!
I'm trying to train a ControlNet conditioning on human pose (openpose 18 keypoints like in your provided demo).
My dataset is DeepFashion, which contains 10K full-body images with long text annotations like these:

"This guy wears a long-sleeve shirt with solid color patterns and a long trousers. The shirt is with cotton fabric. The neckline of the shirt is round. The trousers are with cotton fabric and solid color patterns."
"The upper clothing has medium sleeves, cotton fabric and pure color patterns. It has a lapel neckline. The lower clothing is of long length. The fabric is cotton and it has solid color patterns."

During training, I condition the ControlNet with the above text and the below pose maps obtained from the same Openpose Detector in gradio_pose2image.py

Begin from origin sd2.1. Image size and pose image size are both 512x512
HyperParameter:
batch_size = 8
learning_rate = 1e-5
sd_locked = True
only_mid_control = False
accumulate_grad_batches = 2
precision =32

After training 37 epochs, the result is still poor:

These results fulfill the condition pose and text nicely. But the quality of the face is not as good as your human pose demo.
I have tried using short text prompts by splitting the origin text by "." and randomly choosing one part. But the results didn't get better.
Could you please help me to figure out the reason? Thanks!

Read gradio_pose2image.py first to compare the OpenPose preprocessing used for the demo with the supplied pose maps. Reproduce the reported setup using the listed dataset, prompts, image sizes, and hyperparameters; done means documenting a confirmed cause or a reproducible improvement to the training result.