NVIDIA-NeMo / NVIDIA-NeMo/Curator

DocumentSplitter treats multi-character separators as regex (wrong splits, re.error crash on "++")

Open Beginner friendly
#2,374 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

community-request waiting-on-maintainers
Dominant language
Python
Stars
1.8k
Forks
328
Avg merge
4d 5h
Merged PRs (30d)
30

Description

Description

DocumentSplitter passes its separator to Series.str.split() without regex=False, so pandas treats multi-character separators as regular expressions:

  • DocumentSplitter(separator="..") on text a..b..c produces segments ['', '', '', 'c'] (regex .. matches any two chars) instead of ['a', 'b', 'c']
  • separator="||" splits on empty string alternation — wrong boundaries
  • separator="++" crashes with re.error: nothing to repeat at position 0
  • the documented round-trip with DocumentJoiner is broken, because the joiner always joins with the literal separator
Steps to reproduce
import pandas as pd
from nemo_curator.stages.text.modules import DocumentSplitter

# crash case
DocumentSplitter(separator="++")
# re.error: nothing to repeat at position 0  (raised per batch, at str.split)

# silent corruption case
pd.DataFrame({"text": ["a..b..c"]})["text"].str.split("..")
# [['', '', '', 'c']]
Expected behavior

The separator is documented as a plain string ("The separator to split the documents on") and DocumentJoiner treats it as literal — splitting should too.

Actual behavior

df[self.text_field].str.split(self.separator) — pandas only treats single-char string patterns as literal; multi-char patterns go through re.

Environment

nemo-curator main (93b48525), Python 3.12

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at DocumentSplitter in nemo_curator.stages.text.modules and inspect the pandas Series.str.split call using self.separator. Reproduce the documented cases for "..", "||", and "++", then verify splitting treats the separator literally and preserves the DocumentJoiner round-trip behavior. Add or update regression coverage for these cases and confirm the crash and incorrect boundaries are gone.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.