Reuse engine for multiple consequent runs

Description

Build engines for SDXL.
Then init pipeline. And do several runs. At the first run I get good picture, but the second run gives all grey image.

I've added controlnet and ip-adapter to original code.
init function in my code:

def init():
      pipe = StableDiffusionPipeline(
            pipeline_type=PIPELINE_TYPE.XL_IP,
            ip_adapter_scale=[0.8],
            ip_adapter_path=[IP_ADAPTER_CKPT],
            custom_controlnet_path=CONTROLNET_CKPT,
        **kwargs_init_pipeline)
    
    pipe.loadEngines(
        ENGINE_DIR,
        PYTORCH_MODEL,
        ONNX_DIR,
        **kwargs_load_engine)
    
    _, shared_device_memory = cudart.cudaMalloc(pipe.calculateMaxDeviceMemory())
    pipe.activateEngines(shared_device_memory)

then i have generate function, which uses loaded engines to generate images:

def generate(out_fpath):
    ...
    controlnet_scale = [0.8]
    controlnet_scale = torch.FloatTensor(controlnet_scale)
    demo_kwargs = {'input_image': input_images, 'controlnet_scales': controlnet_scale, 'image_embeds': img_emb}
    args_run_demo = (out_fpath, [prompt], [negative_prompt], SIZE[1], SIZE[0], 1, 1, 0, False)
    pipe.loadResourcesAllocBuf(SIZE[1], SIZE[0], 1, None)
    pipe.run(*args_run_demo, **demo_kwargs)

In main:

    generate('pic1.png')
    generate('pic2.png')

pic1.png is good, but pic2.png is all grey

Seems like I have to reinit something to get correct result, but I don't do that.

In original code there goes pipe.teardown() after generation, but this deletes engines and to make one more call we will need to load them again so it would not be possible to get fast inference.

Help, please, to solve the problem

Environment

TensorRT Version: Tensorrt 10.0.0b6

NVIDIA GPU: A100

NVIDIA Driver Version: 550.54.15

CUDA Version: 11.8

CUDNN Version:

Operating System: ubuntu 20.04

Python Version (if applicable): 3.10

Tensorflow Version (if applicable): -

PyTorch Version (if applicable): 2.2.1

Baremetal or Container (if so, version): -