abetlen / abetlen/llama-cpp-python

Increase server timeout for long requests

Open
#634 3 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

While operating on the CPU, I am able to observe both the request and completion logs. However, due to the time-consuming nature of the request, the server issues a 504 timeout error before the completion is finalized. Therefore, I kindly propose the inclusion of an ENV variable or a flag that would allow for the adjustment of the server timeout duration.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.