PaddlePaddle / PaddlePaddle/FastDeploy

fastdeploy 部署paddleocr,压力测试异常,进程退出

Open
#2,073 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.7k
Forks
756
Avg merge
19h 28m
Merged PRs (30d)
4

Description


温馨提示:根据社区不完全统计,按照模板提问,可以加快回复和解决问题的速度


环境

【FastDeploy版本】: docker镜像 registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
【系统平台】: Linux x64(Ubuntu 22.04)
【硬件】: Nvidia GPU P40, CUDA 11.7 CUDNN 8.4
【编译语言】:Python3.8

问题日志及出现问题的操作流程

  • 附上详细的问题日志有助于快速定位分析
    call ocr
    0# 0x00005650067998A9 in fastdeployserver
    1# 0x00007FAF577AF210 in /usr/lib/x86_64-linux-gnu/libc.so.6
    2# void phi::fusion::ConvFusionKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optionalphi::DenseTensor const&, std::vector<int, std::allocator > const&, std::vector<int, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<int, std::allocator > const&, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, std::vector<int, std::allocator > const&, int, phi::DenseTensor*, std::vector<phi::DenseTensor*, std::allocatorphi::DenseTensor* >) in /opt/fastdeploy/third_libs/install/paddle_inference/paddle/lib/libpaddle_inference.so
    3# void phi::KernelImpl<void ()(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optionalphi::DenseTensor const&, std::vector<int, std::allocator > const&, std::vector<int, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<int, std::allocator > const&, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, std::vector<int, std::allocator > const&, int, phi::DenseTensor, std::vector<phi::DenseTensor*, std::allocatorphi::DenseTensor* >), &(void phi::fusion::ConvFusionKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optionalphi::DenseTensor const&, std::vector<int, std::allocator > const&, std::vector<int, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<int, std::allocator > const&, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, std::vector<int, std::allocator > const&, int, phi::DenseTensor*, std::vector<phi::DenseTensor*, std::allocatorphi::DenseTensor* >))>::KernelCallHelper<paddle::optionalphi::DenseTensor const&, std::vector<int, std::allocator > const&, std::vector<int, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<int, std::allocator > const&, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, std::vector<int, std::allocator > const&, int, phi::DenseTensor*, std::vector<phi::DenseTensor*, std::allocatorphi::DenseTensor* >, phi::TypeTag >::Compute<1, 3, 0, 0, phi::GPUContext const, phi::DenseTensor const, phi::DenseTensor const, phi::DenseTensor const>(phi::KernelContext*, phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&) in /opt/fastdeploy/third_libs/install/paddle_inference/paddle/lib/libpaddle_inference.so
    4# paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, phi::Place const&, paddle::framework::RuntimeContext*) const in /opt/fastdeploy/third_libs/install/paddle_inference/paddle/lib/libpaddle_inference.so
    5# paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, phi::Place const&) const in /opt/fastdeploy/third_libs/install/paddle_inference/paddle/lib/libpaddle_inference.so
    6# paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, phi::Place const&) in /opt/fastdeploy/third_libs/install/paddle_inference/paddle/lib/libpaddle_inference.so
    7# 0x00007FAE3D72085D in /opt/fastdeploy/third_libs/install/paddle_inference/paddle/lib/libpaddle_inference.so
    8# paddle::AnalysisPredictor::ZeroCopyRun() in /opt/fastdeploy/third_libs/install/paddle_inference/paddle/lib/libpaddle_inference.so
    9# fastdeploy::PaddleBackend::Infer(std::vector<fastdeploy::FDTensor, std::allocatorfastdeploy::FDTensor >&, std::vector<fastdeploy::FDTensor, std::allocatorfastdeploy::FDTensor >*, bool) in /opt/fastdeploy/lib/libfastdeploy_runtime.so.0.0.0
    10# fastdeploy::Runtime::Infer() in /opt/fastdeploy/lib/libfastdeploy_runtime.so.0.0.0
    11# 0x00007FAEFC1B0FA4 in /opt/tritonserver/backends/fastdeploy/libtriton_fastdeploy.so
    12# 0x00007FAEFC1B4826 in /opt/tritonserver/backends/fastdeploy/libtriton_fastdeploy.so
    13# TRITONBACKEND_ModelInstanceExecute in /opt/tritonserver/backends/fastdeploy/libtriton_fastdeploy.so
    14# 0x00007FAF5833B83A in /opt/tritonserver/bin/../lib/libtritonserver.so
    15# 0x00007FAF5833C04D in /opt/tritonserver/bin/../lib/libtritonserver.so
    16# 0x00007FAF581F0801 in /opt/tritonserver/bin/../lib/libtritonserver.so
    17# 0x00007FAF58335DC7 in /opt/tritonserver/bin/../lib/libtritonserver.so
    18# 0x00007FAF57B9DDE4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    19# 0x00007FAF5801B609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0
    20# clone in /usr/lib/x86_64-linux-gnu/libc.so.6

[StatusCode.UNAVAILABLE] recvmsg:Connection reset by peer
[StatusCode.UNAVAILABLE] recvmsg:Connection reset by peer
[StatusCode.UNAVAILABLE] recvmsg:Connection reset by peer
[StatusCode.UNAVAILABLE] recvmsg:Connection reset by peer
[StatusCode.UNAVAILABLE] recvmsg:Connection reset by peer

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the OCR call under the reported pressure test in the FastDeploy Triton backend, then inspect the stack from TRITONBACKEND_ModelInstanceExecute through PaddleBackend::Infer and Runtime::Infer. Done means the process remains available during the test and the reported connection resets are isolated to a specific component.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux, python
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.