huggingface / huggingface/optimum-nvidia
Feature request: streamer
Open
enhancement
- Dominant language
- Python
- Stars
- 1k
- Forks
- 103
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
Can you please add support for streamers in the .generate() function?
```
Exception in thread Thread-2 (llm_loop):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "app_llama4.py", line 333, in llm_loop
output = model.generate(**generation_kwargs)
TypeError: TensorRTForCausalLM.generate() got an unexpected keyword argument 'streamer'
```
Contributor guide
Assessment
This issue has not been assessed yet.