NVIDIA-NeMo / NVIDIA-NeMo/Curator
DocumentSplitter treats multi-character separators as regex (wrong splits, re.error crash on "++")
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
Description
DocumentSplitter passes its separator to Series.str.split() without regex=False, so pandas treats multi-character separators as regular expressions:
DocumentSplitter(separator="..")on texta..b..cproduces segments['', '', '', 'c'](regex..matches any two chars) instead of['a', 'b', 'c']separator="||"splits on empty string alternation — wrong boundariesseparator="++"crashes withre.error: nothing to repeat at position 0- the documented round-trip with
DocumentJoineris broken, because the joiner always joins with the literal separator
Steps to reproduce
import pandas as pd
from nemo_curator.stages.text.modules import DocumentSplitter
# crash case
DocumentSplitter(separator="++")
# re.error: nothing to repeat at position 0 (raised per batch, at str.split)
# silent corruption case
pd.DataFrame({"text": ["a..b..c"]})["text"].str.split("..")
# [['', '', '', 'c']]
Expected behavior
The separator is documented as a plain string ("The separator to split the documents on") and DocumentJoiner treats it as literal — splitting should too.
Actual behavior
df[self.text_field].str.split(self.separator) — pandas only treats single-char string patterns as literal; multi-char patterns go through re.
Environment
nemo-curator main (93b48525), Python 3.12
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at DocumentSplitter in nemo_curator.stages.text.modules and inspect the pandas Series.str.split call using self.separator. Reproduce the documented cases for "..", "||", and "++", then verify splitting treats the separator literally and preserves the DocumentJoiner round-trip behavior. Add or update regression coverage for these cases and confirm the crash and incorrect boundaries are gone.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100