microsoft / microsoft/onnxruntime-extensions
StringSplit wrong behaviour when separator is empty
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 479
- Forks
- 144
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 11
Description
According to the documentation in https://www.tensorflow.org/api_docs/python/tf/strings/split:
If sep is None or an empty string, consecutive whitespace are regarded as a single separator, and the result will contain no empty strings at the start or end if the string has leading or trailing whitespace.
However the current implementation performs a character by character split.
https://github.com/microsoft/onnxruntime-extensions/blob/98fb96d4e8c936f6a0013ab9e555fcb115fc60d5/operators/text/string_split.cc#L36-L50
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in operators/text/string_split.cc at the implementation linked in the issue, and compare its empty-separator behavior with the TensorFlow strings.split documentation. The work is done when an empty separator treats consecutive whitespace as one separator and omits empty results at the beginning or end, with the relevant behavior verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100