michaelfeil / michaelfeil/infinity
Disable model warmup
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
### System Info
I am using 0.0.75 and when I use `--no-model-warmup` it still tries to warm up the model.
Therefor CTRL+C for exiting also does not work when warming up.
Maybe that only happens with ONNX. In PyTorch the `--no-model-warmup` does work.
### Information
- [x] Docker + cli
- [x] pip + cli
- [ ] pip + usage of Python interface
### Tasks
- [ ] An officially supported CLI command
- [ ] My own modifications
### Reproduction
1. Use `infinity_emb v2 --port 7997 --no-model-warmup --engine optimum --model-id `
2. Try to use or `curl` to infinity but does not work because of ECONNREFUSED.
3. Try to CTRL+C but does not work. I need to wait until warm up is done.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the command from the issue using the `infinity_emb v2` CLI entry point, `--no-model-warmup`, and `--engine optimum`; compare its behavior with the PyTorch case described. Done means the opted-out CLI does not warm up before serving and CTRL+C can interrupt startup; verify the endpoint no longer remains unavailable during an unnecessary warmup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python, pytorch
- Domain
- cli, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100