Adding two text columns concatenates in pandas and raises here
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 31m
- Merged PRs (30d)
- 640
Description
`pd.Series(["a", "b"]) + pd.Series(["x", "y"])` is `["ax", "by"]`. The same expression in firepanda raises `binary: + is not defined on string`, which #250 records as a test that asserts the refusal rather than leaving the gap unwritten.
Concatenation is the one arithmetic operation a text column has, and it is the only one of the seven that means anything on text. The kernel already has all six comparisons on text, in `firepanda/kernel/text.mojo`, reached through `binary_any` and `binary_value_any` the same way the numeric ones are, so the dispatch and the promotion and the null rule are all in place and what is missing is a loop.
### What pandas does
Measured against 3.0.5 rather than read out of the documentation.
| | |
|---|---|
| `Series(["a", "b"]) + Series(["x", "y"])` | `["ax", "by"]`, dtype str |
| `Series(["a", "b"]) + "x"` | `["ax", "bx"]` |
| `"x" + Series(["a", "b"])` | `["xa", "xb"]`, so the constant carries a side here as it does everywhere else |
| a null on either side | null, not the other side on its own |
| aligning two differently labelled text columns | the union, with a row only one side has coming back null |
| `Series(["a"]) * 3` | `["aaa"]`, because Python's `str` has that operator |
| `Series(["a"]) - Series(["b"])` | `TypeError: unsupported operand type(s) for -: 'str' and 'str'` |
`*` between a text column and an integer is the second operation on that list and it is a different shape from the first, because the two operands have different types and the answer's length depends on a value rather than on a type. It is worth doing in the same change or worth being explicit about leaving out, rather than being discovered later by the conformance suite.
### The shape of the work
A text column is offsets and bytes, so a concatenation cannot write into a buffer whose size it does not know yet. Two passes: one over both sides to sum the pair of lengths per row, which gives the offsets and the total, then one to copy. The first pass is a walk of two offset arrays and no bytes at all, so it is cheap relative to the copy that follows.
The null rule is the usual one and falls out of the shared validity, since a row with either side missing writes a zero length and is marked absent. The constant form writes one side's bytes from the same place every row, so it can hoist the constant's length out of the loop and only the column's offsets vary.
`binary_type` needs a rule saying that `+` on two text types answers a text type. Every other operation on text answers bool, so the existing rule is "text in, bool out" and it grows a case rather than being replaced.
### Done when
- [ ] `+` on two text columns concatenates, in the column form and the constant form with the constant on either side, with the null rule and the alignment matching the table above
- [ ] `binary_type` answers a text type for `+` on two text types, and the lazy engine can still tell an expression's type before a row moves
- [ ] `*` between a text column and an integer repeats, or this issue says in one paragraph why it is somewhere else
- [ ] Tests in `tests/test_text.mojo` with the expected values read off a running pandas, and the refusal test in `tests/test_series_arith.mojo` replaced by an assertion of the answer
- [ ] The differential fuzzer runs `+` on its text columns
### Why this is not in #238
#238 is the fifty callables that were missing from the Python surface, and this is one operation that is missing from a kernel. It was found while writing the tests that close #238's comparison box and it is small enough to stand alone, so it goes here rather than growing that issue a tenth section.
Contributor guide
Research direction
Start in firepanda/kernel/text.mojo, following binary_any, binary_value_any, and binary_type to understand the existing text comparisons and type rules. Read tests/test_text.mojo and tests/test_series_arith.mojo, then run them before changing the behavior. Done means text addition handles both constant sides, nulls, alignment, and type inference, with the integer repetition decision documented or implemented and the differential fuzzer covering text +.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100