PaddlePaddle / PaddlePaddle/FastDeploy

使用C++进行端到端多线程推理时会出现偶发性错误：Mats shapes are not consistent.

Open
#1,857 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.7k
Forks
756
Avg merge
19h 28m
Merged PRs (30d)
4

Description

环境

  • 【FastDeploy版本】： 预编译版：fastdeploy-win-x64-gpu-1.0.5
  • 【系统平台】: Windows x64(Windows10)
  • 【硬件】： Nvidia GPU 3070， CUDA 11.2.2，CUDNN 8.1.1.33
  • 【软件】：vs2019
  • 【编译语言】： C++

问题日志及出现问题的操作流程

可执行文件ocr_infer_end2end.exe参数说明：
argv[1] - 检测模型目录
argv[2] - 识别模型目录
argv[3] - 识别模型字典
argv[4] - 测试图片目录
argv[5] - 图片batchsize  # 调用BatchPredict时输入的图片数
argv[6] - 重复次数 # 使用多线程预测所重复的次数，目的是为了测性能，取单图预测时间的平均值
argv[7] - backend    # 此处使用的是paddleinference推理库，开启GPU
argv[8] - thread_num # 开启的线程数
    • 报错信息
D:\OCR\build\Release>ocr_infer_end2end.exe D:/OCR/model/det D:/OCR/model/rec D:/OCR/model/rec/en_dict.txt D:/OCR/model/imgs/ 2 1000 1 8
 --- load images file done ---
[WARNING] fastdeploy/runtime/runtime_option.cc(376)::fastdeploy::RuntimeOption::SetTrtInputShape        `RuntimeOption::SetTrtInputShape` will be removed in v1.2.0, please use `RuntimeOption.trt_option.SetShape()` instead.
[WARNING] fastdeploy/runtime/runtime_option.cc(376)::fastdeploy::RuntimeOption::SetTrtInputShape        `RuntimeOption::SetTrtInputShape` will be removed in v1.2.0, please use `RuntimeOption.trt_option.SetShape()` instead.
e[1me[35m--- Running analysis [ir_graph_build_pass]e[0m
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0423 10:37:05.709136 11184 executor.cc:186] Old Executor is Running.
e[1me[35m--- Running analysis [ir_analysis_pass]e[0m
e[32m--- Running IR pass [map_op_to_another_pass]e[0m
I0423 10:37:05.730129 11184 fuse_pass_base.cc:59] ---  detected 15 subgraphs
e[32m--- Running IR pass [identity_scale_op_clean_pass]e[0m
e[32m--- Running IR pass [is_test_pass]e[0m
e[32m--- Running IR pass [simplify_with_basic_ops_pass]e[0m
e[32m--- Running IR pass [delete_quant_dequant_linear_op_pass]e[0m
e[32m--- Running IR pass [delete_weight_dequant_linear_op_pass]e[0m
e[32m--- Running IR pass [constant_folding_pass]e[0m
e[32m--- Running IR pass [silu_fuse_pass]e[0m
e[32m--- Running IR pass [conv_bn_fuse_pass]e[0m
I0423 10:37:05.942971 11184 fuse_pass_base.cc:59] ---  detected 48 subgraphs
e[32m--- Running IR pass [conv_eltwiseadd_bn_fuse_pass]e[0m
e[32m--- Running IR pass [embedding_eltwise_layernorm_fuse_pass]e[0m
e[32m--- Running IR pass [multihead_matmul_fuse_pass_v2]e[0m
e[32m--- Running IR pass [vit_attention_fuse_pass]e[0m
e[32m--- Running IR pass [fused_multi_transformer_encoder_pass]e[0m
e[32m--- Running IR pass [fused_multi_transformer_decoder_pass]e[0m
e[32m--- Running IR pass [fused_multi_transformer_encoder_fuse_qkv_pass]e[0m
e[32m--- Running IR pass [fused_multi_transformer_decoder_fuse_qkv_pass]e[0m
e[32m--- Running IR pass [multi_devices_fused_multi_transformer_encoder_pass]e[0m
e[32m--- Running IR pass [multi_devices_fused_multi_transformer_encoder_fuse_qkv_pass]e[0m
e[32m--- Running IR pass [multi_devices_fused_multi_transformer_decoder_fuse_qkv_pass]e[0m
e[32m--- Running IR pass [fuse_multi_transformer_layer_pass]e[0m
e[32m--- Running IR pass [gpu_cpu_squeeze2_matmul_fuse_pass]e[0m
e[32m--- Running IR pass [gpu_cpu_reshape2_matmul_fuse_pass]e[0m
e[32m--- Running IR pass [gpu_cpu_flatten2_matmul_fuse_pass]e[0m
e[32m--- Running IR pass [gpu_cpu_map_matmul_v2_to_mul_pass]e[0m
e[32m--- Running IR pass [gpu_cpu_map_matmul_v2_to_matmul_pass]e[0m
e[32m--- Running IR pass [matmul_scale_fuse_pass]e[0m
e[32m--- Running IR pass [multihead_matmul_fuse_pass_v3]e[0m
e[32m--- Running IR pass [gpu_cpu_map_matmul_to_mul_pass]e[0m
e[32m--- Running IR pass [fc_fuse_pass]e[0m
e[32m--- Running IR pass [fc_elementwise_layernorm_fuse_pass]e[0m
e[32m--- Running IR pass [conv_elementwise_add_act_fuse_pass]e[0m
e[32m--- Running IR pass [conv_elementwise_add2_act_fuse_pass]e[0m
e[32m--- Running IR pass [conv_elementwise_add_fuse_pass]e[0m
I0423 10:37:08.334635 11184 fuse_pass_base.cc:59] ---  detected 43 subgraphs
e[32m--- Running IR pass [transpose_flatten_concat_fuse_pass]e[0m
e[32m--- Running IR pass [auto_mixed_precision_pass]e[0m
e[32m--- Running IR pass [inplace_op_var_pass]e[0m
e[1me[35m--- Running analysis [ir_params_sync_among_devices_pass]e[0m
I0423 10:37:08.344597 11184 ir_params_sync_among_devices_pass.cc:94] Sync params from CPU to GPU
e[1me[35m--- Running analysis [adjust_cudnn_workspace_size_pass]e[0m
e[1me[35m--- Running analysis [inference_op_replace_pass]e[0m
e[1me[35m--- Running analysis [memory_optimize_pass]e[0m
I0423 10:37:08.369583 11184 memory_optimize_pass.cc:220] Cluster name : hardswish_17.tmp_0  size: 1920
I0423 10:37:08.369583 11184 memory_optimize_pass.cc:220] Cluster name : tmp_7  size: 384
I0423 10:37:08.370584 11184 memory_optimize_pass.cc:220] Cluster name : batch_norm_46.tmp_3  size: 1920
I0423 10:37:08.371582 11184 memory_optimize_pass.cc:220] Cluster name : tmp_10  size: 384
I0423 10:37:08.372582 11184 memory_optimize_pass.cc:220] Cluster name : tmp_1  size: 384
I0423 10:37:08.373581 11184 memory_optimize_pass.cc:220] Cluster name : tmp_9  size: 384
I0423 10:37:08.373581 11184 memory_optimize_pass.cc:220] Cluster name : elementwise_add_3  size: 96
I0423 10:37:08.374614 11184 memory_optimize_pass.cc:220] Cluster name : elementwise_add_1  size: 64
I0423 10:37:08.374614 11184 memory_optimize_pass.cc:220] Cluster name : x  size: 12
e[1me[35m--- Running analysis [ir_graph_to_program_pass]e[0m
I0423 10:37:08.420553 11184 analysis_predictor.cc:1404] ======= optimize end =======
I0423 10:37:08.420553 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:08.424551 11184 naive_executor.cc:151] ---  skip [sigmoid_0.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(266)::fastdeploy::Runtime::CreatePaddleBackend     Runtime initialized with Backend::PDINFER in Device::GPU.
e[1me[35m--- Running analysis [ir_graph_build_pass]e[0m
e[1me[35m--- Running analysis [ir_analysis_pass]e[0m
e[32m--- Running IR pass [map_op_to_another_pass]e[0m
I0423 10:37:08.468526 11184 fuse_pass_base.cc:59] ---  detected 14 subgraphs
e[32m--- Running IR pass [identity_scale_op_clean_pass]e[0m
e[32m--- Running IR pass [is_test_pass]e[0m
e[32m--- Running IR pass [simplify_with_basic_ops_pass]e[0m
e[32m--- Running IR pass [delete_quant_dequant_linear_op_pass]e[0m
e[32m--- Running IR pass [delete_weight_dequant_linear_op_pass]e[0m
e[32m--- Running IR pass [constant_folding_pass]e[0m
e[32m--- Running IR pass [silu_fuse_pass]e[0m
e[32m--- Running IR pass [conv_bn_fuse_pass]e[0m
I0423 10:37:08.594483 11184 fuse_pass_base.cc:59] ---  detected 32 subgraphs
e[32m--- Running IR pass [conv_eltwiseadd_bn_fuse_pass]e[0m
e[32m--- Running IR pass [embedding_eltwise_layernorm_fuse_pass]e[0m
e[32m--- Running IR pass [multihead_matmul_fuse_pass_v2]e[0m
e[32m--- Running IR pass [vit_attention_fuse_pass]e[0m
e[32m--- Running IR pass [fused_multi_transformer_encoder_pass]e[0m
e[32m--- Running IR pass [fused_multi_transformer_decoder_pass]e[0m
e[32m--- Running IR pass [fused_multi_transformer_encoder_fuse_qkv_pass]e[0m
e[32m--- Running IR pass [fused_multi_transformer_decoder_fuse_qkv_pass]e[0m
e[32m--- Running IR pass [multi_devices_fused_multi_transformer_encoder_pass]e[0m
e[32m--- Running IR pass [multi_devices_fused_multi_transformer_encoder_fuse_qkv_pass]e[0m
e[32m--- Running IR pass [multi_devices_fused_multi_transformer_decoder_fuse_qkv_pass]e[0m
e[32m--- Running IR pass [fuse_multi_transformer_layer_pass]e[0m
e[32m--- Running IR pass [gpu_cpu_squeeze2_matmul_fuse_pass]e[0m
e[32m--- Running IR pass [gpu_cpu_reshape2_matmul_fuse_pass]e[0m
e[32m--- Running IR pass [gpu_cpu_flatten2_matmul_fuse_pass]e[0m
e[32m--- Running IR pass [gpu_cpu_map_matmul_v2_to_mul_pass]e[0m
I0423 10:37:10.817183 11184 fuse_pass_base.cc:59] ---  detected 9 subgraphs
e[32m--- Running IR pass [gpu_cpu_map_matmul_v2_to_matmul_pass]e[0m
I0423 10:37:10.820181 11184 fuse_pass_base.cc:59] ---  detected 4 subgraphs
e[32m--- Running IR pass [matmul_scale_fuse_pass]e[0m
e[32m--- Running IR pass [multihead_matmul_fuse_pass_v3]e[0m
e[32m--- Running IR pass [gpu_cpu_map_matmul_to_mul_pass]e[0m
e[32m--- Running IR pass [fc_fuse_pass]e[0m
I0423 10:37:10.901134 11184 fuse_pass_base.cc:59] ---  detected 9 subgraphs
e[32m--- Running IR pass [fc_elementwise_layernorm_fuse_pass]e[0m
I0423 10:37:10.917125 11184 fuse_pass_base.cc:59] ---  detected 1 subgraphs
e[32m--- Running IR pass [conv_elementwise_add_act_fuse_pass]e[0m
e[32m--- Running IR pass [conv_elementwise_add2_act_fuse_pass]e[0m
e[32m--- Running IR pass [conv_elementwise_add_fuse_pass]e[0m
I0423 10:37:10.982112 11184 fuse_pass_base.cc:59] ---  detected 34 subgraphs
e[32m--- Running IR pass [transpose_flatten_concat_fuse_pass]e[0m
e[32m--- Running IR pass [auto_mixed_precision_pass]e[0m
e[32m--- Running IR pass [inplace_op_var_pass]e[0m
I0423 10:37:11.008074 11184 fuse_pass_base.cc:59] ---  detected 5 subgraphs
e[1me[35m--- Running analysis [ir_params_sync_among_devices_pass]e[0m
I0423 10:37:11.010123 11184 ir_params_sync_among_devices_pass.cc:94] Sync params from CPU to GPU
e[1me[35m--- Running analysis [adjust_cudnn_workspace_size_pass]e[0m
e[1me[35m--- Running analysis [inference_op_replace_pass]e[0m
e[1me[35m--- Running analysis [memory_optimize_pass]e[0m
I0423 10:37:11.036058 11184 memory_optimize_pass.cc:220] Cluster name : batch_norm_14.tmp_3  size: 6144
I0423 10:37:11.037086 11184 memory_optimize_pass.cc:220] Cluster name : transpose_0.tmp_0  size: 480
I0423 10:37:11.038058 11184 memory_optimize_pass.cc:220] Cluster name : x  size: 576
I0423 10:37:11.039085 11184 memory_optimize_pass.cc:220] Cluster name : batch_norm_15.tmp_4  size: 6144
I0423 10:37:11.039085 11184 memory_optimize_pass.cc:220] Cluster name : tmp_0  size: 480
I0423 10:37:11.040056 11184 memory_optimize_pass.cc:220] Cluster name : batch_norm_17.tmp_4  size: 6144
I0423 10:37:11.041085 11184 memory_optimize_pass.cc:220] Cluster name : tmp_2  size: 480
I0423 10:37:11.041085 11184 memory_optimize_pass.cc:220] Cluster name : matmul_v2_0.tmp_0  size: 32
I0423 10:37:11.042055 11184 memory_optimize_pass.cc:220] Cluster name : shape_1.tmp_0_slice_1  size: 4
I0423 10:37:11.043054 11184 memory_optimize_pass.cc:220] Cluster name : shape_0.tmp_0_slice_1  size: 4
e[1me[35m--- Running analysis [ir_graph_to_program_pass]e[0m
I0423 10:37:11.076035 11184 analysis_predictor.cc:1404] ======= optimize end =======
I0423 10:37:11.077035 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.080062 11184 naive_executor.cc:151] ---  skip [softmax_2.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(266)::fastdeploy::Runtime::CreatePaddleBackend     Runtime initialized with Backend::PDINFER in Device::GPU.
model initialized done
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.084030 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.086053 11184 naive_executor.cc:151] ---  skip [sigmoid_0.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.088033 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.090057 11184 naive_executor.cc:151] ---  skip [softmax_2.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.092041 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.096055 11184 naive_executor.cc:151] ---  skip [sigmoid_0.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.098023 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.100021 11184 naive_executor.cc:151] ---  skip [softmax_2.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.103058 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.106048 11184 naive_executor.cc:151] ---  skip [sigmoid_0.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.108031 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.110045 11184 naive_executor.cc:151] ---  skip [softmax_2.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.112058 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.114042 11184 naive_executor.cc:151] ---  skip [sigmoid_0.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.117012 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.119035 11184 naive_executor.cc:151] ---  skip [softmax_2.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.121042 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.123034 11184 naive_executor.cc:151] ---  skip [sigmoid_0.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.125018 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.129006 11184 naive_executor.cc:151] ---  skip [softmax_2.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.131006 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.134027 11184 naive_executor.cc:151] ---  skip [sigmoid_0.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.137001 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.139024 11184 naive_executor.cc:151] ---  skip [softmax_2.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.140998 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.144021 11184 naive_executor.cc:151] ---  skip [sigmoid_0.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.145995 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.148018 11184 naive_executor.cc:151] ---  skip [softmax_2.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.150993 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.153017 11184 naive_executor.cc:151] ---  skip [sigmoid_0.tmp_0], fetch -> fetch
[INFO] fastdeploy/runtime/runtime.cc(364)::fastdeploy::Runtime::Clone   Runtime Clone with Backend:: Backend::PDINFER in Device::GPU.
I0423 10:37:11.155010 11184 naive_executor.cc:151] ---  skip [feed], feed -> x
I0423 10:37:11.158012 11184 naive_executor.cc:151] ---  skip [softmax_2.tmp_0], fetch -> fetch
assignThreadImg done
W0423 10:37:11.203963 11184 gpu_resources.cc:85] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 11.4, Runtime API Version: 11.2
W0423 10:37:11.211957 11184 gpu_resources.cc:115] device: 0, cuDNN Version: 8.1.
W0423 10:37:11.218953 11184 gpu_resources.cc:241] WARNING: device:  . The installed Paddle is compiled with CUDNN 8.2, but CUDNN version in your machine is 8.1, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version.
warmup 1 times done
warmup 2 times done
warmup 3 times done
warmup 4 times done
warmup 5 times done
warmup 6 times done
warmup 7 times done
warmup 8 times done
warmup 9 times done
warmup 10 times done
  ***** infer time per image = 24.1037ms *****
  ***** repeat 1 times done *****
  ***** infer time per image = 23.5881ms *****
  ***** repeat 2 times done *****
  ***** infer time per image = 23.9566ms *****
  ***** repeat 3 times done *****
  ***** infer time per image = 23.2061ms *****
  ***** repeat 4 times done *****
[ERROR] fastdeploy/vision/common/processors/mat_batch.cc(33)::fastdeploy::vision::FDMatBatch::Tensor    Mats shapes are not consistent.
    • 源码
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

#include <thread>
#include "fastdeploy/vision.h"
#include <direct.h>
#include <sys/stat.h>
#include <iostream>
#include <fstream>
#include <io.h>

#ifdef WIN32
const char sep = '\\';
#else
const char sep = '/';
#endif


void loadImgFiles(const std::string& imgPath, const std::string& imgFormat, std::vector<std::string>& imgs)
{
  intptr_t hFile = 0;
  struct _finddata_t fileInfo;
  std::string str, fileFormatName;
  if (0 != strcmp(imgFormat.c_str(), ""))
  {
    fileFormatName = "\\*." + imgFormat;
  }
  else
  {
    fileFormatName = "\\*";
  }
  if ((hFile = _findfirst(str.assign(imgPath).append(fileFormatName).c_str(), &fileInfo)) != -1)
  {
    do
    {
      imgs.push_back(str.assign(imgPath).append("\\").append(fileInfo.name));
    } while (_findnext(hFile, &fileInfo) == 0);
    _findclose(hFile);
  }
}


void assignThreadImg(std::vector<std::vector<cv::Mat>>* thread_images, std::vector<cv::Mat> all_images, int thread_num)
{
  // number of image files in images folder
  size_t count = all_images.size();
  size_t num = count / thread_num;
  for (int i = 0; i < thread_num; i++)
  {
    std::vector<cv::Mat> temp_list;
    if (i == thread_num - 1) {
      for (size_t j = i * num; j < count; j++)
      {
        temp_list.push_back(all_images[j]);
      }
    }
    else
    {
      for (size_t j = 0; j < num; j++)
      {
        temp_list.push_back(all_images[i * num + j]);
      }
    }
    thread_images->push_back(temp_list);
  }
}


void assignBatchImg(const std::vector<cv::Mat>& imgPerGpu, std::vector<std::vector<cv::Mat>>& imgByBatch, int max_batch_size)
{
  imgByBatch.clear();
  int imgNum = imgPerGpu.size();
  const int maxBatchTimes = imgNum / max_batch_size;
  if (maxBatchTimes == 0)//如果待处理图片总数不足一个batch，则一次性处理所有图片
  {
    std::vector<cv::Mat> images;
    for (int i = 0; i < imgNum; i++)
    {
      images.push_back(imgPerGpu[i]);
    }
    if (!images.empty()) {
      imgByBatch.push_back(images);
    }
  }
  else
  {
    const int rest = imgNum - maxBatchTimes * max_batch_size;
    for (int i = 0; i < maxBatchTimes; i++)
    {
      std::vector<cv::Mat> images;
      for (int j = 0; j < max_batch_size; j++)
      {
        images.push_back(imgPerGpu[i * max_batch_size + j]);
      }
      if (!images.empty()) {
        imgByBatch.push_back(images);
      }
    }
    if (rest != 0)
    {
      std::vector<cv::Mat> images;
      for (int i = maxBatchTimes * max_batch_size; i < maxBatchTimes * max_batch_size + rest; i++)
      {
        images.push_back(imgPerGpu[i]);
      }
      if (!images.empty()) {
        imgByBatch.push_back(images);
      }
    }
  }
}

double getTime() {
  return (double)cv::getTickCount();
}
double getTimeDiff(double t2, double t1) {
  double diff = t2 - t1;
  return diff * 1000.0 / cv::getTickFrequency(); // ms
}


void Predict(fastdeploy::pipeline::PPOCRv3* model, int batchsize, std::vector<cv::Mat>& images) {
  std::vector<std::vector<cv::Mat>> m_batchImg;
  assignBatchImg(images, m_batchImg, batchsize);

  for (auto const& image_file : m_batchImg) {
    std::vector<fastdeploy::vision::OCRResult> res;
    if (!model->BatchPredict(image_file, &res)) {
      std::cerr << "Failed to predict." << std::endl;
      return;
    }
    /*for (auto &r : res)
    {
      std::cout << r.Str() << std::endl;
    }*/
  }
  
}


void InitAndInfer(const std::string& det_model_dir, const std::string& rec_model_dir, 
                  const std::string& rec_label_file, std::vector<cv::Mat> imgs, const fastdeploy::RuntimeOption& option, 
                  int thread_num, int batchsize, int repeatTimes) {
  auto det_model_file = det_model_dir + sep + "inference.pdmodel";
  auto det_params_file = det_model_dir + sep + "inference.pdiparams";

  auto rec_model_file = rec_model_dir + sep + "inference.pdmodel";
  auto rec_params_file = rec_model_dir + sep + "inference.pdiparams";

  auto det_option = option;
  auto rec_option = option;

  // The cls and rec model can inference a batch of images now.
  // User could initialize the inference batch size and set them after create PP-OCR model.
  int rec_batch_size = 6;

  // If use TRT backend, the dynamic shape will be set as follow.
  // We recommend that users set the length and height of the detection model to a multiple of 32.
  // We also recommend that users set the Trt input shape as follow.
  det_option.SetTrtInputShape("x", { 1, 3, 64,64 }, { 1, 3, 640, 640 },
    { 1, 3, 960, 960 });
  rec_option.SetTrtInputShape("x", { 1, 3, 48, 10 }, { rec_batch_size, 3, 48, 320 },
    { rec_batch_size, 3, 48, 2304 });

  // Users could save TRT cache file to disk as follow. 
  // det_option.SetTrtCacheFile(det_model_dir + sep + "det_trt_cache.trt");
  // cls_option.SetTrtCacheFile(cls_model_dir + sep + "cls_trt_cache.trt");
  // rec_option.SetTrtCacheFile(rec_model_dir + sep + "rec_trt_cache.trt");

  auto det_model = fastdeploy::vision::ocr::DBDetector(det_model_file, det_params_file, det_option);
  auto rec_model = fastdeploy::vision::ocr::Recognizer(rec_model_file, rec_params_file, rec_label_file, rec_option);

  assert(det_model.Initialized());
  assert(rec_model.Initialized());

  // The classification model is optional, so the PP-OCR can also be connected in series as follows
  // auto ppocr_v3 = fastdeploy::pipeline::PPOCRv3(&det_model, &rec_model);
  auto ppocr_v3 = fastdeploy::pipeline::PPOCRv3(&det_model, &rec_model);

  // Set inference batch size for cls model and rec model, the value could be -1 and 1 to positive infinity.
  // When inference batch size is set to -1, it means that the inference batch size 
  // of the cls and rec models will be the same as the number of boxes detected by the det model. 
  ppocr_v3.SetRecBatchSize(rec_batch_size);

  if (!ppocr_v3.Initialized()) {
    std::cerr << "Failed to initialize PP-OCR." << std::endl;
    return;
  }
  std::cout << "model initialized done" << std::endl;

  std::vector<decltype(ppocr_v3.Clone())> models;
  for (int i = 0; i < thread_num; ++i) {
    models.emplace_back(std::move(ppocr_v3.Clone()));
  }

  std::vector<std::vector<cv::Mat>> thread_images;
  assignThreadImg(&thread_images, imgs, thread_num);
  std::cout << "assignThreadImg done" << std::endl;

  // warmup model
  std::vector<std::vector<cv::Mat>> warmup_imgs;
  std::vector<fastdeploy::vision::OCRResult> res;
  auto warmupTimes = 10;
  assignBatchImg(imgs, warmup_imgs, batchsize);
  for (auto i = 0; i < warmupTimes; i++) {
    for (auto modelIndex = 0; modelIndex < thread_num; modelIndex++)
    {
      for (auto& images : warmup_imgs) {
        if (!models[modelIndex].get()->BatchPredict(images, &res)) {
          std::cerr << "Failed to predict." << std::endl;
        }
      }
    }
    std::cout << "warmup " << i + 1 << " times done" << std::endl;
  }

  for (int i = 0; i < repeatTimes; i++)
  {
    auto startTtime = getTime();
    std::vector<std::thread> threads;

    for (int i = 0; i < thread_num; ++i) {
      threads.emplace_back(Predict, models[i].get(), batchsize, thread_images[i]);
    }

    for (int i = 0; i < thread_num; ++i) {
      threads[i].join();
    }
    auto endTime = getTime();
    auto inferTime = getTimeDiff(endTime, startTtime);
    std::cout << "  ***** infer time per image = " << inferTime / imgs.size() << "ms *****  " << std::endl;
    std::cout << "  ***** repeat " << i + 1 << " times done *****  " << std::endl;
  }

  
}

int main(int argc, char* argv[]) {
  fastdeploy::RuntimeOption option;
  int flag = std::atoi(argv[7]);

  if (flag == 0) {
    option.UseCpu();
  }
  else if (flag == 1) {
    // use paddleinference
    option.UseGpu();
    option.paddle_infer_option.enable_log_info = true;
    option.UsePaddleBackend();

  }
  else if (flag == 2) {
    option.UseGpu();
    option.UseTrtBackend();
  }
  else if (flag == 3) {
    option.UseGpu();
    option.UseTrtBackend();
    option.EnablePaddleTrtCollectShape();
    option.EnablePaddleToTrt();
  }
  else if (flag == 4) {
    option.UseKunlunXin();
  }

  std::string det_model_dir = argv[1];
  std::string rec_model_dir = argv[2];
  std::string rec_label_file = argv[3];
  std::string image_file_path = argv[4];
  int batchsize = std::atoi(argv[5]);
  int repeatTimes = std::atoi(argv[6]);
  int thread_num = std::atoi(argv[8]);

  std::vector<std::string> flist;
  loadImgFiles(image_file_path, "jpg", flist);
  std::cout << " --- load images file done --- " << std::endl;

  std::vector<cv::Mat> imgs;
  for (auto i = 0; i < flist.size(); ++i)
  {
    auto img = cv::imread(flist[i]);
    imgs.push_back(std::move(img));
  }

  InitAndInfer(det_model_dir, rec_model_dir, rec_label_file, imgs, option, thread_num, batchsize, repeatTimes);
  return 0;
}
  • 【多线程推理出现偶发性错误】
    • 在 multi_thread_ocr.cc多线程预测代码的基础上进行修改，当重复执行多线程预测时有时会出现报错[ERROR] fastdeploy/vision/common/processors/mat_batch.cc(33)::fastdeploy::vision::FDMatBatch::Tensor Mats shapes are not consistent.导致程序停止，该问题出现概率较高。源代码可直接编译运行，输入图像分辨均为2048*2048。请教为何会出现这类问题，以及如何解决？

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with ocr_infer_end2end.exe using the documented model, image, batch-size, repetition, backend, and thread-count arguments. Start by examining the multithreaded BatchPredict path and the Runtime::Clone calls shown in the log. Done means repeated concurrent inference no longer produces the “Mats shapes are not consistent” error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.