huggingface / huggingface/optimum-nvidia

Feature request: streamer

Open
#60 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1k
Forks
103
Avg merge
2m
Merged PRs (30d)
1

Description

Can you please add support for streamers in the .generate() function?

```
Exception in thread Thread-2 (llm_loop):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "app_llama4.py", line 333, in llm_loop
output = model.generate(**generation_kwargs)
TypeError: TensorRTForCausalLM.generate() got an unexpected keyword argument 'streamer'
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.