deepspeedai / deepspeedai/DeepSpeed
[BUG] inference --prompt special inputs
@mrwyattii is already working on this.
Since Dec 13, 2023.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
When running in inference, one of the input parameters is a sentence which is given after --prompt flag.
As all arguments are parsed inside inference to build the "shell" command, prompt argument is wrapped in quotation marks
--prompt.
The wrapping quotations were ' and lately I have changed it to " (which solves a lot of issues where the prompt has a ' inside of it).
But with the current solution, there is a wrong behavior (all inner " should be escaped, other wise the terminal ignores them) where we have inner quotations inside the prompt and there is a - inside those inner quotes.
For example:
parameters of any command
--model_name_or_path tiiuae/falcon-40b --max_new_tokens 128 --prompt "I'm feeling hot he said " temperatures is -4C outside""
user args after parsing:
['--model_name_or_path', 'tiiuae/falcon-40b', '--max_new_tokens', '128', '--prompt', "I'm feeling hot he said ", 'temp', 'is', '-4C', 'outside']
issues:
- inner quotations are dropped because of using format on the string inside the python
- the fact that split according to white spaces and new arguments are identified by the - sign isn't right, because here -4C will be identified as a new argument that leads to:
unrecognized arguments: -4C outside
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.