How to process requests with FastChat api parallelly or in a batch style?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
I have to get vector embedding /do sentimental analysis for **millions of senteces**. Currently, it seems that I can only process sentence one by one with RESTful API Server launched in the background. It is very slow ad much GPU memory is unoccupied. Is there any method so that I can process senteces with FastChat api parallelly or in a batch style?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is identified. Begin at the RESTful API server entry point and determine whether the requested parallel or batch handling is supported; done would require a clearly specified implementation or documentation outcome plus validation for the stated workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100