M6: pandas parity, first tier

Tracking issue for **M6**. Specification: everything tagged `(M6)` in [`06-pandas-parity.md`](../blob/main/docs/specs/06-pandas-parity.md), and the counted version of it in [`08-m6.md`](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/08-m6.md).

The largest milestone by volume and the smallest by risk. It parallelizes well across contributors, and it is the natural place for a second person to start.

### What changed in this issue

This was seven scope bullets. It is now fifteen issues with a measured target on each, and the change is not that M6 got bigger. The pandas surface was counted rather than remembered: pandas 3.0.3 is 1413 public names across 21 namespaces, 1125 public callables, 3267 parameters on those callables, 46 error types and 70 option keys, measured by `pixi run surface` in [firepanda-compat](https://github.com/tamnd/firepanda-compat) against the installed pandas and committed as a JSON file so a pandas upgrade shows up as a diff.

M6 covers roughly 530 of those 1125 callables, which is 47 percent of pandas in one milestone. Seven bullets was not enough structure to hold that, and more to the point a bullet is ticked by a person who believes the thing works while a target is measured by a suite. One line of the old checklist read `rolling with window, min_periods, center, closed, step`, which is one checkbox over five parameters whose interactions are the entire difficulty of the feature, and `rolling` in pandas 3.0 actually takes eight.

### Scope, in order

Each item is an issue with a conformance target rather than a description of doneness. The target is the L3 rate over that workstream's callables, reported by `pixi run conformance` and enforced by the CI ratchet. L3 means every parameter takes every one of its values and the combinations that interact are enumerated, and the five levels are defined in [`01-what-100-percent-means.md`](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/01-what-100-percent-means.md).

- [ ] #152 J. The 46 error and warning types, L4 available on every operation that can fail
- [ ] #153 I. The cheap surface, 40 callables, L1 and L3 both 100 percent
- [ ] #154 A1. The `Index` type, 73 callables, L3 90 percent
- [ ] #155 A2. `MultiIndex`, 86 callables, L3 70 percent
- [ ] #156 A3. `loc`, `iloc`, `at`, `iat`, `xs` and the methods that need an index, L3 100 percent
- [ ] #495 A4. `DatetimeIndex`, 144 members, L3 90 percent
- [ ] #157 B1. The 49 string methods that need no regex, L3 100 percent
- [ ] #158 B2. RE2 and the eight pattern methods, L3 100 percent with the lookaround and backreference divergences registered
- [ ] #159 C. Categoricals, 8 callables plus the dtype and the `observed` paths, L3 100 percent
- [ ] #160 D. Nested data, `.list`, `.struct`, `explode`, `json_normalize`, L3 100 percent, with anything past the pandas five reported as an extension rather than as conformance
- [ ] #161 E. Windows, 53 callables, L3 90 percent, `win_type` allowed to trail
- [ ] #162 F. Reshaping, 16 callables, L3 100 percent
- [ ] #163 G. GroupBy completion, 32 callables, L3 90 percent
- [ ] #164 H. Statistics and the rest of the frame, 61 callables, L3 90 percent
- [ ] #165 K. `api.types` predicates and the option keys, L3 100 percent on the predicates and on the eight load bearing options
- [ ] #270 L. The four column types three conformance sections are waiting on: the timestamp, the duration, the dictionary and the nested pair, plus the `dt` accessor. Added after the fact and it is the largest item on this list, because 228 of the 334 failures on the board never ran an operation at all. Four of the five are done, in #272, #275, #278 and #282, and the board has fallen 334 to 94
- [ ] #287 M. The `dt` accessor, 42 names on a column that can be read and not asked. The fifth item of #270 lifted out because it is a namespace rather than a column type, specified in [`14-the-dt-accessor.md`](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/14-the-dt-accessor.md). 68 cases and 108 runs, the largest single measured item left on the board
- [x] #238 L. Arithmetic and comparison, 50 callables, L3 90 percent, with `level` allowed to trail

The order is argued in [`08-m6.md`](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/08-m6.md) and the short version is this. J goes first because retrofitting error types across 500 callables is a task that never actually happens, and it is the smallest piece of work that lifts L4 across the whole library at once. I goes second because 40 names at ten minutes each is the best day in the milestone and nobody will ever prioritise `rtruediv` on its own. A goes third because everything leans on it: `groupby(...).agg(...)`, `value_counts`, `describe` and `unstack` all produce indexed results, and building those against a frame type with no index means building them twice. B goes fourth and ships whole, split into the 49 methods that need no regex and the eight that do, so the regex engine does not block the rest of the namespace. C, D, E, F, G and H are independent once A exists. K is last because it only matters once there is a Python front door to hand a frame through. L was not in the original fourteen and is not new scope either: it is the M1 operators, which nobody noticed were missing until the conformance suite ran against the Python surface for the first time and reported them as 100 of its 167 failures, and it can be picked up as early as anyone wants it because it depends on nothing that is not already there and because a boolean mask is unreachable without it.

### The instrument

Every checkbox above is closed by a number, and the numbers come from [firepanda-compat](https://github.com/tamnd/firepanda-compat), which is being built alongside this milestone rather than after it. Its own progress belongs here because none of the exit criteria below can be evaluated until it exists.

The full log of what has landed, one entry per piece of work in the order it happened, is [`docs/m6-progress.md`](https://github.com/tamnd/firepanda/blob/main/docs/m6-progress.md). It was in this issue body until the body reached the size GitHub refuses to store. The five most recent entries are below and the other 122 are in the file.

- [x] One format for a whole float column, in [#734](https://github.com/tamnd/firepanda/pull/734) against [#733](https://github.com/tamnd/firepanda/issues/733) and [#8](https://github.com/tamnd/firepanda/issues/8), with document 62 beside it and compat [#154](https://github.com/tamnd/firepanda-compat/pull/154) scoring it. The last thing named as left over in document 61 section 9 was that pandas picks one format for a whole float column, and picking it per value turned out to be the cause of three separate looking bugs. `1e-5` printed as `1e-05` where pandas prints `0.00001`. A column of 1234567.125 and 2.0 printed the second as `2.0` where pandas prints `2.000`, and there is no rule you can apply to 2.0 on its own that produces that, because the three zeros are there for the number beside it. And a column holding one enormous value now sends every value in it to scientific notation, which is why the tall corpus frame prints 2.644949e+15 like pandas rather than 2644949145455244.0. The model underneath is four steps and every one of them was measured. Everything is written fixed at six places first. Trailing zeros then come off the whole column at once, one place at a time, and only while every plain number in it still ends in a zero, with one place always put back so a column of whole numbers reads 1.0 and 2.0. The column then measures its longest cell counting the leading place from the slice before this one and goes to scientific notation only if that is over twelve characters and something in the column is over a million, which is both halves together and is why 123456789.0 stays fixed and 1234567890.0 does not, and why 1234567.125 stays fixed and 1234567.0625 does not, two values that differ only in their fraction. Separately, a nonzero value smaller than the last place printed sends the column to an exponent with no length test at all, because writing it fixed would print a different number, which puts the boundary at exactly 1e-6. Three plausible models were wrong before the fourth survived, and the one that kept looking right and kept being wrong was that this is a magnitude threshold. Three consequences are worth stating because they are testable and were tested. The rows that will not be printed are cut before the format is chosen, so a huge value in the elided middle of a hundred row column cannot change what the printed values look like. A NaN, an infinity and a missing cell are not numbers for any of it, so they neither block the stripping nor get stripped nor count towards the length. And the labels down the side are a column too and go through the same code, so a float level goes to an exponent as a whole exactly as a column of values does, while each column of a frame decides on its own and a frame can print one column in scientific notation beside one in fixed point. Rounding is half to even the way C's printf is, so 0.0078125 at six places is 0.007812 and 2.5 at no places is 2 while 3.5 is 4, all four of which the obvious implementation gets wrong. Twelve new Mojo tests, eleven new Python tests that compare whole renderings against a running pandas rather than against a literal, and two compat cases over four runs, taking cases from 3400 to 3402 and runs from 4483 to 4487 with L2 at 172 of 1125, L3 at 101 and the nine failing runs unchanged. Two things in the implementation are equivalences rather than rules and are named as such in document 62 section 10, which is that a magnitude at or above 1e15 takes the scientific branch without being written fixed first and that a null cell is left out of the length measurement, and both are provably unable to change an answer at the default precision. What is left of the printing group is the two spellings, `` against `NaN` and `string` against `str`, neither of which this renderer decides.
- [x] The last thing left in the printing group, settled by keeping the difference rather than removing it: a cell with nothing in it prints `` here and, most of the time, `NaN` in pandas. The reason it stays is that pandas does not have a spelling for a missing value, it has four, and which one a reader gets depends on the dtype. Measured over nine one column frames with a value in row zero and nothing in row one, float64, str and category print `NaN`, object prints `None`, datetime64 and timedelta64 print `NaT`, and Int64, boolean and float64[pyarrow] print ``. The split is not arbitrary. `NaN` is what a numpy float array gives you because there is nowhere else in a float array to record absence, `NaT` is the same trick for a timestamp where the sentinel is a particular integer, `None` is what an array of Python objects gives you, and `` is what pandas prints for its own nullable types, where absence lives in a mask beside the values rather than smuggled into them. Every firepanda column is an Arrow array with a validity bitmap, so the bottom row of that table is the closest thing pandas has to a firepanda column, and `` is pandas' own word for it rather than a spelling this library invented. The difference is also not confined to printing, which is the part that decides it: on the corpus frame float64_half_null, whose float column holds a computed NaN in row zero and a hole in row one, reading the first three rows out gives `[nan, nan, -inf]` in pandas and `[nan, None, -inf]` here, because pandas cannot tell the two apart by the time the column exists and firepanda can. Printing them the same would make the rendering say something about the column that is not true, which is the one thing a renderer must not do, and that is the same rule document 62 followed pointed at the value rather than at the layout. The cost is real and is why this could not be waved through as cosmetic: `` is one character wider than `NaN`, the width of a column is the width of its widest cell, so a single missing value moves every other value in that column one place and two otherwise identical renderings diff on every line rather than on the line with the gap in it. That width is why the earlier printing cases stepped around null bearing frames, and with the difference registered they no longer have to. What the entry does not cover is named rather than left implicit: `isna` agrees on both sides including on a genuine NaN, count, dropna, fillna and every skipping reduction agree, and a missing value comparing as missing rather than as False is `engine/comparison-null`, which predates all of this. Document 63 in seven sections ([firepanda #739](https://github.com/tamnd/firepanda/pull/739)) and the twenty ninth registry entry, `engine/missing-spelling`, with four divergent cases over a float column's rendering, the same column read out as Python values, a text column and a timestamp column, plus a control case outside its patterns asserting that `isna` still agrees and is therefore required to pass ([compat #155](https://github.com/tamnd/firepanda-compat/pull/155)). Cases 3402 to 3407 and runs 4487 to 4492, with the nine failing runs unchanged
- [x] The five `str` names that are about case, `upper`, `lower`, `isspace`, `islower` and `isupper`, and then the same five again against the right pandas. The first version was measured against pandas holding its text as Python objects and passed every test written for it, and the conformance board failed it on the unicode frame the first time it was scored: `strings/upper` gave `SS` where pandas gives a single capital sharp s, and `strings/lower` gave a dotted i where pandas gives a plain one. The reason is that pandas 3 gives a text column an Arrow backed dtype by default, and `str.upper` and `str.lower` on such a column are Arrow compute kernels rather than Python's own string methods. Arrow uses the simple case mappings, which are one code point in and one code point out, so a row that goes in eight characters long comes out eight characters long whatever is in it. Python uses the full mappings. Measured on pandas 3.0.5 with pyarrow 25.0.1, the default dtype raises `straße` to `STRAẞE` and the object dtype raises it to `STRASSE`, the default lowers `İstanbul` to `istanbul` and the object dtype lowers it to a dotted i followed by a combining dot, the default lowers a final sigma to a plain sigma and the object dtype gives the final form, and `ª` is lower to the object dtype and not to the default. The board scores the default, so Arrow is what this library has to answer, and the two rows of the unicode frame that hold a sharp s and a dotted capital I are the only place in the whole corpus where the two pandas backends visibly disagree. They caught a defect every local test had passed, which is the clearest argument yet for a corpus frame carrying awkward text rather than convenient text. The size of the difference was then measured rather than guessed at, by walking all 1111998 code points through both the Mojo standard library and pyarrow. The mapping half is 149 code points: 39 where the library expands, which is Python's answer rather than Arrow's, including the sharp s, the ligatures, the Greek iota subscripts and the letters written with a preceding apostrophe, and 110 the library has never heard of because they were added to Unicode after its copy of the data was made, including the whole Garay block. 149 is small enough to carry as a list, so `firepanda/kernel/casefix.mojo` is that list with Arrow's answer in each direction, generated by `tools/gen_casefix.py` and committed rather than built, because the generator needs pyarrow and the Mojo build does not have it. It costs nothing on the common path: a pass over the bytes of an element looking for any byte at or above the lead byte of the lowest corrected code point rules out every ASCII element without decoding anything, a binary search per code point rules out ordinary accented text so that `éèê` still goes through the library in one call, and only an element that really holds one of the 149 is built a code point at a time. The predicate half was left alone on purpose, because it is 1384 code points rather than 149, 17 on isspace and 845 on islower and 557 on isupper, which is category data rather than a patch list and means firepanda carrying its own generated Unicode tables. That is worth doing and is named as the follow up rather than smuggled in here, and three of the differences are asserted as tripwires in the Python tests so that a move on either side fails a test rather than passing quietly. Documents [64](https://github.com/tamnd/firepanda/blob/main/docs/specs/64-the-case-of-a-letter.md) in eight sections ([firepanda #741](https://github.com/tamnd/firepanda/pull/741) and [#746](https://github.com/tamnd/firepanda/pull/746)), twenty runs armed on the board with `upper` and `lower` passing on all four string frames and the three questions passing on three of them ([compat #156](https://github.com/tamnd/firepanda-compat/pull/156)). A missing row asked what case it is written in answers a missing value here and False in pandas, which is the disagreement `startswith` and `endswith` already had, so the three questions went into `engine/string-predicate-null` rather than into an entry of their own and the registry stays at 29. Board 2540 of 4492, up 20, with 115 divergent and the same nine failing runs
- [x] `str.capitalize` and `str.swapcase`, the two names of the case group the Mojo standard library has no call for, so both are walked a character at a time here and neither needed any case data past the 149 corrections `upper` and `lower` landed with a day earlier. Capitalising is the first character raised and every other character dropped, which is pandas' rule and not the word by word thing the name suggests, so a row of several words comes back with one capital in it and a row starting with a digit comes back unchanged. Swapping case has to know which case a character is already in and deliberately does not ask: `islower` and `isupper` are the two names this library is still wrong about for 1384 code points, so a kernel over them would have inherited every one of those, and the mappings answer the same question and are corrected, since a character that lowers to something else was upper and one that raises to something else was lower. The thirty one titlecase characters are the only ones the mappings cannot classify, because they are in neither case and Arrow leaves them exactly as they are though both of their mappings would move them, so `casefix.mojo` grew a fourth table `KEPT_BY_SWAP` and the generator derives it from Arrow rather than from a list somebody typed. An ASCII row skips the walk and flips one bit per letter. `tools/gen_casefix.py` now checks both rules itself and refuses to write a table if either stops holding, and both names were run against a live pandas over all 1112064 code points on their own and over sixty thousand random words with no row differing. Document [64](https://github.com/tamnd/firepanda/blob/main/docs/specs/64-the-case-of-a-letter.md) grew two sections and is now ten ([firepanda #750](https://github.com/tamnd/firepanda/pull/750)), ten runs armed on the board with both names passing on all four string frames and the two matching signature cases passing ([compat #157](https://github.com/tamnd/firepanda-compat/pull/157)), no divergence to register so the registry stays at 29. `title` is the name of this shape that is not here, because its word boundary needs to know whether a character is cased at all and that rule disagrees with Arrow on 1295 code points, which is the category table [#748](https://github.com/tamnd/firepanda/issues/748) is about. `casefold` is next and is a different animal, since pyarrow has no casefold kernel and pandas falls back to Python for it whatever dtype the column has, so it is the one name in this group where the full mappings are right and a row can come back longer than it went in. Board 2552 of 4492, up 12, with 115 divergent and the same nine failing runs
- [x] `str.casefold`, which looks like a third case and is not one. Nobody writes text in it and nobody reads it, and its one promise is that two rows a reader would call the same come out as the same bytes, so `Straße` and `STRASSE` both fold to `strasse` where lowering them says the two are different. The price of that promise is that a folded row can be longer than the row that went in, which is the exact opposite of the rule the `upper` and `lower` slice spent its length on, and the reason that is not an inconsistency anybody chose is the finding of this one: pyarrow has no casefold kernel at all, so a pandas text column falls back to Python's own `str.casefold` for this one method while every other case method stays in Arrow. Both pandas backends agree here as a result, which is true of no other name in the group, and the full mappings are the right answer for this name and the wrong answer for the four beside it. Following Arrow for `upper` and Python for `casefold` is copying what pandas actually answers rather than picking a side and being consistent about it. `firepanda/kernel/casefold.mojo` is the 353 code points that fold to something other than their lower case, 104 of them to more than one and none to more than three, which makes it a difference table rather than a copy of the folding database: everything else in Unicode folds to exactly what it lowers to, so the kernel asks this table first and falls through to the corrected lower case path for the rest, which is the 149 corrections earning their keep a third time. There is no byte prefilter on this path and there cannot be one, which is the one piece of the earlier work that did not carry over. `upper` and `lower` rule an element out by looking for a byte at or above 0xC3, since the lowest code point they correct is U+00DF, and a good deal of ordinary accented text leaves on that test alone. The lowest code point here is the micro sign at U+00B5, whose lead byte is 0xC2, and 0xC2 is the lowest lead byte any non ASCII character can have, so asking whether an element could hold one of these and asking whether it is not ASCII are the same question, and the kernel asks the second because it is the cheaper spelling of it. Two rows carry the difference and both are tested: `İstanbul` folds to a plain i, a combining dot and then `stanbul`, longer than it went in, where the same row lowered by Arrow is `istanbul` with no dot at all, and `DŽ`, `Dž` and `dž` all fold to the last of them where `swapcase` leaves the middle one exactly as it found it. Verified against a live pandas over all 1112064 code points on their own and over sixty thousand random words with no row differing, and `tools/gen_casefold.py` asserts that folding a row is folding its characters one at a time before it will write a table. Document [64](https://github.com/tamnd/firepanda/blob/main/docs/specs/64-the-case-of-a-letter.md) grew a section and is now eleven ([firepanda #755](https://github.com/tamnd/firepanda/pull/755)), five runs armed on the board ([compat #158](https://github.com/tamnd/firepanda-compat/pull/158)), no divergence to register so the registry stays at 29. `title` is now the only name of the case group still missing, and it waits on the Unicode category tables of [#748](https://github.com/tamnd/firepanda/issues/748) rather than on an afternoon. Board 2558 of 4492, up 6, with 115 divergent and the same nine failing runs
- [x] The four character classes `isspace`, `islower` and `isupper` are answered out of, which is the table [#748](https://github.com/tamnd/firepanda/issues/748) named and the one the case mapping work deliberately left alone. The three questions had been scored on all four string frames and passing since they landed, and they were wrong about 1384 code points the whole time, because the corpus held no row any of those code points appears in. Correct on a corpus and correct are different states and a board cannot tell them apart, which is the finding of this piece of work rather than a footnote to it. [#748](https://github.com/tamnd/firepanda/issues/748) proposed generating the tables from the published Unicode data files and that is not what happened: pandas answers these three names out of Arrow, so Arrow is not an approximation of the right answer here, it is the right answer, and a table built from the Unicode files would be a second opinion to reconcile with the first every time utf8proc and whichever release we happened to read drifted apart. `tools/gen_charclass.py` reads pyarrow directly, which makes the generator twenty lines rather than a parser for a file format and makes a code point unable to disagree. The issue also warned the tables would be megabytes and they are not: Unicode hands out properties in blocks, so lower is 2326 code points in 667 runs, upper is 1928 in 657, titlecase is 31 in 10 and space is 29 in 10, and all four together are 2688 numbers and under eleven kilobytes. A class is held as its runs written flat, a start then one past an end then the next start, so membership is the parity of a lower bound search with no second comparison of the kind a hit or miss table needs, and below 128 each class is two 64 bit words and no search at all. The three row level rules were measured rather than assumed and the generator refuses to write a table unless pyarrow confirms them over every code point and sixty thousand random words: a row is whitespace when it is not empty and every character is a space, a row is lower case when at least one character is lower and none is upper or titlecase, and upper the same way round. The titlecase characters are their own class because Arrow calls them neither case where the standard library called them both, and the 31 of them turn out to be exactly the `KEPT_BY_SWAP` list `swapcase` already derived from the mappings, which is asserted by a test rather than claimed in prose. All three now agree with pandas on every code point in Unicode, asserted as a sweep rather than a sample, and the test that used to name three rows where the two differed is that sweep. Document [65](https://github.com/tamnd/firepanda/blob/main/docs/specs/65-the-classes-a-character-belongs-to.md) in nine sections ([firepanda #759](https://github.com/tamnd/firepanda/pull/759)). Nothing on the board moved and that is the point of the entry. Board 2558 of 4492, unchanged, with 115 divergent and the same nine failing runs
- [x] `str.title`, `str.istitle` and `str.isascii`, the last three names of the case group, and the four corpus rows that should have been there a month ago. The two title names are the first in this part of the library whose answer is not decidable one code point at a time, because a word boundary is a fact about what came before rather than about the character in hand, and neither of them needed a new table. Whether a character is cased is the three classes of [#759](https://github.com/tamnd/firepanda/pull/759) ORed together, and the mapping needed nothing either, because `utf8_title` equals `utf8_upper` for every code point in Unicode and is never longer than one character, measured rather than assumed. That has a consequence a reader should see coming and will not: `dž` at the start of a word becomes `DŽ`, the whole capital, and not `Dž`, the titlecase form that exists for exactly this purpose, because Arrow does not use it. The word rule is not the one the name suggests either. A word does not end at whitespace, it ends at the first character in no case at all, so a digit and an apostrophe both start a new one, `don't` titles to `Don'T`, `abc1def` titles to `Abc1Def`, and `A1b` is not titled while `A1B` is. `isascii` is the odd one out twice over: pyarrow has no `utf8_is_ascii` kernel at all though `pandas.Series.str.isascii` works, so there is nothing for the two sides to disagree about, and it is the only question in the group a row of nothing answers yes to, since it asks what a row does not contain. All three are exact against a live pandas over every code point in Unicode, and the two title names are exact over all 2401 arrangements of four characters drawn from an alphabet chosen to be the seven this rule has anything different to say about. Document [65](https://github.com/tamnd/firepanda/blob/main/docs/specs/65-the-classes-a-character-belongs-to.md) grew two sections and is now ten, document 64 section 9 stops saying `title` is missing ([firepanda #761](https://github.com/tamnd/firepanda/pull/761)). On the board, three arms and four rows ([compat #159](https://github.com/tamnd/firepanda-compat/pull/159)): the unicode frame goes from twelve rows to sixteen and gains a titlecase letter, a kra, a non breaking space and a name with an apostrophe and a digit in it, which between them are one row for each question the frame could not previously tell apart. Every case reading that frame was rerun, the oracle stayed perfect at 4492 of 4492, the relaxation sweep stayed load bearing on all nineteen declarations, and nothing moved except the three new names, so the rows cost nothing and would have caught the defect that shipped. `istitle` and `isascii` joined `engine/string-predicate-null` rather than getting entries of their own, so the registry stays at 29. Board 2574 of 4492, up 16, with 117 divergent and the same nine failing runs. Twenty five `str` names are left and most of them want a regex engine
- [x] `str.isalpha`, `str.isnumeric`, `str.isdigit`, `str.isdecimal` and `str.isalnum`, the last five names of the accessor that ask what a character is, and the two corpus rows that tell the three number questions apart. Four more classes of exactly the kind [#759](https://github.com/tamnd/firepanda/pull/759) built: alphabetic is 145672 code points in 684 runs, numeric is 1924 in 146, digit is 1685 in 136 and decimal is 770 in 72, and all eight classes together are 4764 numbers and under twenty kilobytes. `isalnum` needs no class at all, because a character is alphanumeric exactly when it is alphabetic or numeric, and the generator asserts that identity, the emptiness of the overlap between the two, and the nesting of decimal inside digit inside numeric against pyarrow over every code point before it writes anything, because a subset relation nobody checked is how a table gets one entry wrong for a decade. The rule these five share is the simple one the case questions are not, which is that the row has a character in it and every character it has is in the class, and that was measured too rather than assumed by analogy: over every code point and sixty thousand words drawn from a pool weighted so the alphabetic class does not drown out the other seven, since it is 145672 against under 2400 for every other class and an unweighted draw measures one rule eight times. The finding here is not the names, it is that `pandas.Series(["½"]).str.isdigit()` is True and `"½".isdigit()` is False, on the same character, in the same process. Arrow calls anything written as a single number sign a digit and Python reserves the name for something narrower, which is 877 code points of difference, more than the whole of Python's digit class. That is a rule difference rather than a version gap, and pandas answers Arrow whichever way the column is held, so there is no dtype that reconciles the two the way there is for `str.upper`. The test writes all three answers out side by side rather than asserting only the one implemented, because a test that asserts a half sign is a digit and nothing else reads like a typo to the next person. The version gap is separately visible: Arrow knows 8946 letters Python does not. All five are exact against a live pandas over every code point in Unicode and over every arrangement of three characters drawn from an alphabet chosen to cross the classes. Document [65](https://github.com/tamnd/firepanda/blob/main/docs/specs/65-the-classes-a-character-belongs-to.md) grew two sections and is now twelve, document 64 section 9 stops saying anything is missing ([firepanda #764](https://github.com/tamnd/firepanda/pull/764)). On the board, five arms and two rows ([compat #160](https://github.com/tamnd/firepanda-compat/pull/160)). The five cases were already registered and carried a note claiming the unicode frame held a character that answered the three number questions differently, and measured across all four string frames the whole corpus produced three answer profiles and none of them told a decimal digit from a digit or a digit from a numeric one, so an arm answering all three out of one table would have passed every run. This is the first time document 34's habit was applied before shipping rather than after and the answer was still no, which is the more useful half of the lesson. A half sign and a Roman eight close it and every answer combination the three questions can give now has a row behind it. The five joined `engine/string-predicate-null` on the same terms as the case questions, so the registry stays at 29 and every question the accessor asks about a row is now inside that one entry. Board 2599 of 4492, up 25, with 122 divergent and the same nine failing runs. Nothing in the accessor that asks what a character is is missing now, and the twenty five `str` names left are about patterns, splitting and joining
- [x] `str.contains`, `str.match`, `str.fullmatch` and `str.count`, the first four names of the accessor that ask about a pattern rather than about a character, shipped on a smaller promise than pandas makes and an exact one. pandas reads the argument to all four as a regular expression and there is no regular expression engine here, so the choice was to skip the four until there is one or to find a class of patterns where the absence of an engine cannot be detected. There is one: a pattern holding none of the twelve characters `.^$*+?{}[]\|()` matches exactly the characters it is written with, in that order, and nothing else, so `re.search("bc", row)` and `"bc" in row` are the same question for every row there is. The Python layer reads the pattern, a literal one goes to a kernel and gets pandas' answer, and one holding a metacharacter is refused with a message naming the character that caused it. The alternative worth ruling out in writing is searching for the metacharacter literally and saying nothing, because `contains(".")` under that reading keeps nearly no rows where pandas keeps nearly all of them, and a caller who ported a program would find their filter had quietly emptied while the board read a wrong answer rather than a gap. `case=False` and a non zero `flags` are refused on the same grounds, since an ignored argument is the one failure mode a compatibility layer exists to prevent. Three of the four cost nothing to implement: contains is contains, `match` is starts with and `fullmatch` is equality, and `pattern.mojo` had all three already because a `LIKE` pattern is read into exactly those shapes, so `str.match` and `str.startswith` are one kernel call with two names and the differences pandas keeps between them are not about bytes. Only the count needed a kernel and it is the search in a loop with the cursor moved past each hit, so matches do not overlap and `count("aa")` on four a's is two, which is what a regular expression engine scanning for successive matches answers. The finding is the empty pattern. `pandas.Series(["héllo"], dtype="str").str.count("")` is 7 and `len(re.findall("", "héllo"))` is 6, on a row of five characters and six bytes, because Arrow counts an empty match at every byte offset and once past the end and pandas 3 holds text in Arrow. Measured across all three ways pandas can hold the column, object answers 6 and both `str` and `ArrowDtype` answer 7, so this is not a rule anybody chose, it is an implementation detail of one backend that became the default backend, and `str.count("")` changed its answer when the default dtype changed and changed it to a number of bytes in a method whose other answers are numbers of matches. This library answers 7 for the same reason it calls a half sign a digit, and all three answers are written side by side in the test so the assertion does not read like a typo. Document [66](https://github.com/tamnd/firepanda/blob/main/docs/specs/66-a-pattern-that-is-only-itself.md) in eight sections ([firepanda #767](https://github.com/tamnd/firepanda/pull/767)), six arms and three new cases on the board ([compat #161](https://github.com/tamnd/firepanda-compat/pull/161)). The three new cases are a gap in the board rather than in the library and are worth naming as such: the registry had `strings/match` and `strings/fullmatch` and both used a regular expression, so there was no literal form of either question anywhere in the corpus and an engine that answers literals exactly could not be scored on either name. The corpus check that the last two slices both needed was run again before anything was armed and came back yes without changes, since the pattern `a` produces seven distinct profiles of the four answers across the four string frames and `foo` produces five. Two registry entries grow and no new one is added: contains, match and fullmatch join the twelve cases in `engine/string-predicate-null`, and count joins `len`, `find` and `rfind` in `engine/string-count-width`, because how many times a pattern appears in a row that is not there is the same unanswerable question those three already ask. The registry stays at 29. Board 2622 of 4502, up 23, with 126 divergent and the same nine failing runs. Sixteen `str` names are left, and the shape of the remaining work is now clear rather than a list: eight of them are about cutting a row up or putting rows together and several answer a column of lists, which is a type this library does not have and is the real blocker rather than the pattern, four need the engine, `normalize` needs the Unicode normalization tables, and `replace` is the near one because its `regex` argument defaults to False in pandas 3
- [x] `str.replace` with a literal pattern, the fifth name of the accessor that asks about a pattern and the only one of the five that narrows nothing, because pandas 3 defaults its `regex` argument to False and so the ordinary call is a literal replacement already. The four names a day earlier all shipped on a smaller promise than pandas makes; this one needs no promise, and the metacharacter check those four use only comes into play when a caller writes `regex=True` and asks for an engine on purpose. What it cost instead is a kernel shape. `text_replace` is the first kernel in `pattern.mojo` whose answer is text, and a mask or a number has a size known before the search runs, which is what lets every other kernel there allocate its column up front and fill it across morsels in parallel. How long a row comes out of a replace is not known until the search has run on that row, so this one builds into a `StringBuilder` in a serial loop, which is the shape `text_repeat` already had. `n` works at every sign it can have, negative meaning every match and zero handing the row back without searching it and a positive number meaning that many counted from the left, matches do not overlap so `replace("aa", "X")` on four a's is `XX`, a mapping `pat` applies its pairs one after another with each reading the last one's output and is unpacked in the Python layer so the crossing still takes two strings, and a missing row stays missing. The finding is a stronger version of the one the four landed with. An empty pattern is counted in characters here and in bytes by `count`, on the same row, in the same accessor, and `replace("", "-")` on `héllo` inserts six dashes while `count("")` on it answers seven. There is no rule behind the difference. `pyarrow.compute.replace_substring` does not terminate on an empty pattern, which is [apache/arrow#39149](https://github.com/apache/arrow/issues/39149) and which was measured here rather than inferred, since a single row holding two ASCII characters was still running after forty five seconds at a steady twenty nine percent of a core while a non empty pattern on the same column returned instantly, and pandas carries a guard for that one case in `_arrow_string_mixins.py` with the issue number written into the comment, falling back to Python's own `str.replace`, which counts characters, while `count` has no guard because `count_substring` terminates and so stays in Arrow, which counts bytes. Both numbers are pandas' numbers so both are given, and the Python test asserts the two next to each other in one function on purpose because either one alone reads like a bug. Three of the four refusals are pandas' own or the missing engine, but `case=False` is a real narrowing and is named as such: pandas honours it on this method by escaping the pattern, adding `re.IGNORECASE` and running the object path, measured on `["Abc", "abc", "ABC"]` coming back `['Xbc', 'Xbc', 'XBC']`, and this library refuses it rather than ignoring it so the board reads a name that does not resolve instead of a wrong answer. The case folding search it wants is now owed to five places, since `contains`, `match` and `fullmatch` refuse the same argument and `casefold` already has the mapping half solved, and what is missing is a search that folds both sides as it goes rather than folding the column into a copy, which would be correct and would double the memory the accessor touches. Document [67](https://github.com/tamnd/firepanda/blob/main/docs/specs/67-the-one-pattern-that-needs-no-promise.md) in eight sections ([firepanda #774](https://github.com/tamnd/firepanda/pull/774)), three arms over nine runs and every one of them passing including on the null heavy frame ([compat #162](https://github.com/tamnd/firepanda-compat/pull/162)), which makes this the first name in the whole accessor to arrive with nothing at all to register, since every previous one carried the missing row divergence and this one answers text where a missing row stays missing on both sides. The registry stays at 29. Two corpus changes are the board's own gap rather than the library's and are disclosed the way the three the day before were: `strings/replace-empty` is new so the unit split is two rows of a table and not only a paragraph in a document, and `strings/replace-n` was given a second frame because it ran on `strings_ascii` alone and that frame is the alphabet cut at every length from nothing up to twenty letters, so no row of it holds the letter `a` twice and the case whose entire reason for existing is the limit could not have failed if the limit were ignored outright. Board 2632 of 4505, up 10, with 126 divergent and the same nine failing runs. The strings section stays at 14 of 58 callables at L3, because `strings/replace-regex` and `strings/replace-backreference` are cases of this same callable and both want the engine. Fifteen `str` names are left in three piles: eight want a column of lists, which is now plainly the largest thing between this accessor and the end of it and has nothing to do with text at all, four want the engine, `normalize` wants the Unicode normalization tables, and `decode`, `encode` and `translate` are three small separate problems of which `translate` is the nearest, being a per character mapping over a table the caller supplies, which is the same hash table shape the label half of `rename` has been waiting on since document 50
- [x] `str.translate`, which reads like `str.replace` with several patterns at once and is not that at all. Two things separate them and both matter. A key here is always exactly one character, so nothing is ever searched for and no match can overlap another, where `replace` has to find a pattern of any length and then decide what happens to the bytes after it. And every key is applied in the same pass, where a mapping given to `replace` is several replacements each reading the last one's output, so the table that sends `a` to `b` and `b` to `a` really swaps them here and turns every one of them into `a` there. That second difference is why this needed a kernel of its own rather than a loop over the one that landed the day before: one pass with a lookup per character is not a faster spelling of running the table entry by entry, it is a different answer, and it is the answer pandas gives. The rule itself is the first in a while with no argument in it about which of pandas' three string backends is answering, because pandas hands the table straight to Python's own `str.translate`, which does `table[ord(character)]` and leaves the character alone whenever that raises a `LookupError`, so there is no pyarrow kernel in the path and nothing to measure a backend split against. Everything was measured against pandas 3.0.5 anyway and everything agreed with reading Python's own documentation, which is not something the four names before this could say. A key is a code point ordinal and a value is an ordinal, a string of any length, or `None`, and `None` and the empty string are the same request, which is why the crossing carries no third case for a delete and the Python side turns one spelling into the other before the kernel ever sees it. What a key maps to is never looked at again, so `{ord("a"): "aa"}` on `ab` is `aab` and not a loop. A key that is not an integer, or is negative, or is at or above `0x110000`, never matches anything, and pandas does not refuse such a table, it simply does half of what it looks like it does, so `{"a": "X", ord("b"): "Y"}` replaces the `b` and silently ignores the `a`, and this drops those keys to match rather than refusing them, which is the exact behaviour and not a lenient reading of it. A value out of range and a value of another type are refused with pandas' own two sentences reproduced word for word, because a caller who has just read one of them should not have to learn a second phrasing. The one narrowing is the table itself. pandas takes anything with a `__getitem__`, since it only ever subscripts the thing, so a list works and a string works and a user written class works, and this takes a mapping and refuses the rest. The reason is the crossing: serving the general case means holding a Python object and asking it about every character of every row, a Python call per character, and avoiding exactly that is what going into a kernel is for, so ten entries against a million rows would be a million calls back into the interpreter to answer what a direct array answers in a single load. It is a deliberate choice about which kind of honest the board should see, since a refusal reads as a gap, which is true, and an answer at interpreter speed would read as a pass, which would also be true and would hide that the fast path had been abandoned. `str.maketrans` builds a mapping in all four of its forms and is how anybody actually writes a table, so the refused shapes are the unusual ones. The kernel keeps two seats and every key lands in exactly one: a direct array of 128 slots indexed by the ordinal, and an ordered list of the keys above 127 with a binary search over it, which is the same split `_in_class` already makes in the same file for the same reason, plus a byte loop for the case where the table has no wide key and the row has no non ASCII byte, because then a byte is a character and the decode buys nothing. It is serial rather than split across morsels for the reason `replace` was, that the output length is not known before the kernel runs. The scalar twin is deliberately stupid, walking the table from the front for every character with no seats and no ordering requirement, and both are run on every case in the Mojo tests and asserted to agree before either answer is checked, so a disagreement points at the seats and not at the rule. This is also the one name in the accessor with a door of its own, and the reason is not the shape of its answer, which is text like twenty two others, it is that its argument is a table and not a scalar and there is no way to fold a table into a string because a replacement may hold any character and so no character is left to separate one entry from the next, which is why it crosses as two columns for exactly the reason `is_in` already takes one. Document 07 warns about a door per method and the rule those three doors follow is about answer shape, so this sits outside that rule rather than being an exception to it, and there is only one argument in the namespace that is column sized. Document [68](https://github.com/tamnd/firepanda/blob/main/docs/specs/68-a-table-is-not-a-small-replace.md) in eight sections ([firepanda #778](https://github.com/tamnd/firepanda/pull/778)), two arms over six runs and every one passing ([compat #163](https://github.com/tamnd/firepanda-compat/pull/163)). Nothing went into the divergence registry, which is the second name in a row with nothing to register, and section 2 is the reason: with no Arrow in the path there was no second opinion for pandas and this library to disagree about. The registry stays at 29. One corpus change is disclosed the way the last two slices' were and is the board's gap rather than the library's: `strings/translate` was already in the corpus unanswered and it maps `abc` onto `xyz`, none of which is itself a key, so an implementation running the table entry by entry would give exactly the same answer on all three frames and the case could never have tested the one thing that makes this name different from `replace`, which is why `strings/translate-swap` is new and maps `a` to `b` and `b` to `a`. The separation check was run before arming and both cases come back distinguishable from every other case in the section on all three frames. Board L3 102 of 1125 with the strings section at 15 of 58 callables, up one, and the ratchet floor raised with it. Fourteen `str` names are left and the piles have not changed: eight want a column of lists, which is now plainly more than half of what remains and has nothing to do with text, four want the regular expression engine, `normalize` wants the Unicode normalization tables, and `encode` and `decode` are two small separate problems. The case folding search is still owed to five places. And the label half of `rename` wants the same shape this just built, a table of keys and values with a lookup per element and one pass and no rescanning, so when that kernel is written it should cross the same way
- [x] `case=False` on `str.contains`, `str.match`, `str.fullmatch` and `str.replace`, which is one argument that four names had been refusing since the first of them landed and which turned out to have no written answer behind it. pandas documents `case` as "if True, case sensitive" and stops there, so what folding actually means had to be recovered by measurement rather than read. It is not `str.casefold`, which is the guess Unicode itself invites: pandas answers these three out of `pyarrow.compute.match_substring(ignore_case=True)` and that is simple case folding, one code point to one code point with no expansions, so `STRASSE` does not contain `straße` and `FIANCE` does not contain `fiance` where full folding says both do. It is not lowercasing either, and four separate characters say so, since final sigma, the micro sign, the long s and the Kelvin sign all fold onto another letter and all stay themselves under `lower`. Simple folding is a real Unicode table that Python does not expose, so rather than vendor `CaseFolding.txt` and let its version drift away from whatever Arrow was built against, the table is derived from what Python does expose and then checked against Arrow directly: fold to the full casefold where that is a single code point, which is 1426 of them, otherwise to the lowercase where that is a single code point different from the original, which is 28 more and is the branch that gets capital sharp s down to small sharp s, otherwise leave it alone, which is 76 that fold to themselves, plus three pairs the rule gets wrong that are hardcoded after being found by hand. That is 1457 entries with 1427 distinct targets, generated by `tools/gen_searchfold.py` into a comptime table, and the check that makes it trustworthy is a sweep of all 1.1 million code points asking Arrow whether it agrees with the rule's answer for each one, which reported zero wrong in 91.8 seconds, so the table is not an approximation of Arrow's behaviour, it is Arrow's behaviour for every input that exists. The fourth name is answered by a different language entirely and was checked rather than assumed: pandas' `_str_replace` raises `NotImplementedError` when `case` is false, the accessor falls back to the object path, and that path builds `re.escape(pat)` and compiles it with `re.IGNORECASE`, so one of the four goes through Python's regular expression engine and three go through a C++ kernel, and an exhaustive comparison says the two folds agree everywhere, which means one table serves all four. The pattern is folded once before the search and the column is never folded at all, because folding a column into a copy would double the memory the accessor touches and a fold that changes a character's byte length would then make every offset wrong, so the kernel decodes the row a code point at a time and folds as it goes, which is also why a match can cover a different number of bytes than the pattern does. On the Mojo side the fold is a second dispatch word rather than a seventh argument, `contains_folded` beside `contains`, on the `strip` and `strip_chars` precedent, and the practical payoff of that is that `tools/bindings.py` and the generated binding needed no change at all for a slice touching four public methods. The finding is pandas disagreeing with itself, found by sweeping the argument space rather than by testing interesting values: `n=0` means no replacements and `n=0, case=False` means all of them, in the same method on the same column, because the Arrow path takes the number at its word and the `re.sub` fallback treats a count of zero as unlimited, which is the seam between two implementations showing through and which nobody designed. Compatibility is the goal so this library does both, widening the zero to minus one in the Python layer when the search folds so the kernel's contract stays honest and the inconsistency lives in one place with a comment on it, and it is on the list to file upstream. Document [69](https://github.com/tamnd/firepanda/blob/main/docs/specs/69-the-fold-a-search-is-not-the-fold-a-reader-gets.md) in nine sections ([firepanda #783](https://github.com/tamnd/firepanda/pull/783)), eight arms and a new corpus frame ([compat #164](https://github.com/tamnd/firepanda-compat/pull/164)). The corpus change is the fourth slice running where document 34's habit found the board weaker than it looked, and this time it was weaker in two ways at once. `strings/contains-case-false` was already sitting there unarmed with the pattern `A`, and an ASCII pattern cannot separate any two case insensitive rules, since simple folding and full casefolding and plain lowercasing agree on every ASCII input on every row always, so the case would have gone green the moment it was armed and would have gone green over a kernel using the wrong table. Worse, no other frame could have caught it either, because fifty six frames held not one row where the three rules disagree. `strings_folding` is twenty seven rows chosen so each rule answers differently on some of them, deliberately with no null in it so a failure never needs a second look to decide which of two things it was about, and measured afterwards the pattern long s separates the search fold from casefolding by two rows and from lowercasing by seven while the pattern `ss` separates it from casefolding by five, which is why three of the eight new cases exist only to carry a pattern other than `A`. Three of the eight are divergent rather than passing and they join `engine/string-predicate-null`, which needed no new argument, because folding the comparison changes which rows match and changes nothing about a row there is nothing to compare, so that entry now names nine cases. The registry stays at 29 for the third slice running. Board L3 102 of 1125 with the strings section holding 15 callables and parameter coverage up to 54.3 percent, and the ratchet says nothing went backwards. Fourteen `str` names are left and the piles are unchanged except that one of them is now priced: eight want a column of lists, `normalize` wants the Unicode normalization tables, `encode` and `decode` are two small separate problems, and the regular expression engine is blocking four names rather than one, which is the first time its absence has been worth counting
- [x] `str.partition` and `str.rpartition`, the first two names of the accessor whose answer is a frame rather than a column, which is the whole reason they were picked next out of the fourteen still missing. Each cuts every row at one occurrence of a separator and hands back what came before it, the separator itself, and what came after, and everything else about them is a substring search this library has had since document 66, so they force the question of what an accessor does with three columns without also needing the regular expression engine, a list column type or the Unicode normalization tables. The difference between the two names is not only the one the names suggest: cutting at the first occurrence rather than the last is half of it, and the other half is a row the separator is not in at all, which survives whole and goes into the first column for `partition` and the third for `rpartition`, which is Python's rule and is the one thing an implementation written from the name alone gets wrong while passing everything else anybody would think to check. The habit of the last four documents was to work out which of pandas' backends is answering, and here there is nothing to work out, which is itself the finding: `pyarrow.compute` has `partition_nth_indices`, which is about sorting, and the four split kernels, and no partition kernel at all, so `ArrowExtensionArray._str_partition` is `self._apply_elementwise(lambda val: val.partition(sep))`, a Python loop calling CPython's own method on every row, and the object backend does the same thing through `_str_map`, which makes the oracle for this slice Python rather than Arrow for the first time in five documents. `text_partition` searches once per row and fills three builders from that one offset rather than answering one column at a time and running the search three times to keep a third of each answer, since the search is the entire cost here and the slicing after it is free. It hands back a `List` and not a `Tuple`, which reads oddly and is not a style choice: taking a Mojo tuple apart copies what comes out of it and a `StringArray` does not conform to `ImplicitlyCopyable`, so the list goes all the way from the kernel through `chars_partition` and `py/text.partition` to the Python layer where the three are moved out one at a time, and that shape was arrived at by writing the tuple version first and watching three separate call sites refuse to compile. `partition` gets a function of its own in `firepanda/py/text.mojo`, which is the second door added in three slices and is not the drift document 07 warns about, because the rule those doors follow is that they are picked by the shape of the answer and never by the shape of the argument, and three columns is a shape no door carried, where `translate` beside it genuinely is an exception and was argued for as one because what makes it different is a column sized argument. Two things do not match pandas and both come from pandas doing this a row at a time. The columns are labelled `"0"`, `"1"` and `"2"` where pandas uses the integers 0, 1 and 2, because a firepanda frame holds text column labels and an Arrow schema names its fields with strings, and every value under them agrees. And pandas decides both of its refusals and the width of its answer from the rows it actually reaches, so measured on 3.0.5 `pd.Series([], dtype="str").str.partition("")` raises nothing and answers a frame with no columns at all, a column holding only missing rows answers a frame with one, and only a column with something readable in it raises `ValueError: empty separator`, none of which is the three columns the documentation for this name describes. This library checks the separator once before it starts and always answers three columns, and both measurements are asserted in the Python test file so the disagreement is recorded as a decision rather than as an oversight. `sep=""` and a separator that is not a string are refused with CPython's own two sentences behind the `firepanda:` prefix, and `expand=False` is refused by name because it wants one column of three element tuples and there is no column type for one, which is the same wall `split`, `rsplit`, `findall`, `join` and `extractall` are behind and which comes down for all six at once or not at all. Document [70](https://github.com/tamnd/firepanda/blob/main/docs/specs/70-an-answer-that-is-wider-than-a-column.md) in nine sections ([firepanda #787](https://github.com/tamnd/firepanda/pull/787)), eight arms and a new registry entry ([compat #165](https://github.com/tamnd/firepanda-compat/pull/165)). The corpus check document 34's habit calls for was run before anything was armed and found the board weaker than it looked for the fifth slice running: no row of `strings_pattern` holds a hyphen and a hyphen is the separator both existing cases use, so neither of them ever exercised a found separator and an implementation that always answered not found would have passed both. They were kept as they are rather than repaired, because a row the separator is not in is exactly where the two names differ in the way nobody guesses and the pair is the only thing on the board that says so, and six new cases use `o` instead, which `foofoobar` holds three times and which is therefore the separator that can tell the two searches apart. Those six read one column out by position and rename it so what they compare is the values, and `strings/partition-null` on the null heavy frame asks three questions at once, that a missing row is missing in all three columns rather than empty in any of them, that an empty row is not a missing one, and that a separator at the very start leaves the first column empty rather than dropping the row. `engine/integer-column-labels` is the thirtieth registry entry, breaking three slices of the registry staying at 29, and it is the first entry about a label rather than about a value. It is asserted in two places, the three columns a partition hands back and a frame built from a dictionary with integer keys, the second of which reaches the same difference with no string method in the way and is what says this belongs to frames rather than to the accessor, with a control that reads those three columns by position and passes. The same difference will be visible from `str.split(expand=True)` and from `str.extract` when those arrive, so what closes it is a column label type rather than anything any of those methods could write differently. Board L3 102 of 1125 with the strings section at 15 of 58 callables and parameter coverage at 54.3 percent, six new passes and two new divergences, and the ratchet says nothing went backwards. Twelve `str` names are left and the piles are unchanged except that the largest one is now smaller by two: `cat`, `wrap`, `normalize` and `get_dummies` need no new column type and are the next ones reachable, `join`, `findall`, `extractall` and the default form of `split` and `rsplit` want a list or tuple column, the regular expression engine is blocking four names, `normalize` wants the Unicode normalization tables, and `encode` and `decode` want a binary column
- [x] `str.cat` in the form that has no `others`, which folds a whole text column into one string and is the first name on this accessor whose answer is narrower than a column rather than wider. It is a function of its own in `firepanda/py/text.mojo` for the same reason `partition` was one slice ago, and saying that twice in two consecutive slices is the point: the rule those doors follow is that the shape of the answer picks the door, three columns is a shape and one string is a shape, and neither of them is a column, so two decisions that look opposite are the same decision. `translate` remains the only genuine exception in that file, and it is one because what makes it different is the shape of its argument. pandas spells two different operations `str.cat` and picks between them by whether `others` was passed, and the two share a name, a `sep` and a `na_rep` and nothing else: with `others` it lines a second column up against this one and concatenates row by row and the answer is a column as tall as the input, and without it the answer is a scalar, a missing row is dropped rather than propagated, and the separator goes between rows rather than between the two pieces of a row. The rule worth knowing about the half that is written is that a missing row is dropped rather than blanked and dropped takes its separator with it, so `["a", None, "b"]` joined by `-` is `a-b` and not `a--b`, which is what a join written the obvious way produces because the obvious way replaces the row with an empty string and leaves the separator standing. An empty row is not a missing one, it is readable and never dropped and it does leave two separators with nothing between them, so `["a", "", "b"]` joined by `-` is `a--b` and the same output is reached by opposite rules, which is exactly the situation where checking one of them proves nothing and is why there is a test on each side of the crossing that puts the pair side by side. `na_rep` turns the first case into the second, and pandas reads whether to drop off whether the argument was supplied rather than off what it holds, which makes `na_rep=""` a real request rather than a way of spelling the default: it keeps the row and its separator and drops only the text, where leaving the argument out drops both. The crossing into Mojo therefore carries the stand in and the flag as two arguments, because an implementation that read the flag back out of the string would collapse two different answers into one. `text_join` counts the bytes in one pass and writes them in another, since a join is the one text operation whose answer length is known exactly before anything is written and the alternative appends into a buffer that copies everything written so far every time it doubles, which for a tall column is the entire cost of the operation, and the twin in `scalar.mojo` is the growing version so that what the two are checked against each other on is the separator count, which is the part of a join that is wrong by one when it is wrong at all. The refusals here are the first in this series that are not pandas' own sentences, because pandas checks neither `sep` nor `na_rep` and lets both fall into `str.join`, so measured on 3.0.5 `sep=1` comes back as `AttributeError: 'int' object has no attribute 'join'` and `na_rep=1` as `TypeError: sequence item 2: expected str instance, int found`, both of which name pandas' implementation rather than the caller's mistake and the second of which leaks a row index that depends on the data, so repeating them would mean repeating an accident and both are refused with a message naming the argument while the tests assert the error type on both libraries rather than the sentence. `join` is dropped unread, because it says how to line `others` up and there is no `others` to line up, and refusing a value pandas accepts for an argument that does nothing would be a divergence invented for tidiness; pandas does not validate it either and takes `join="bogus"` without a word even when there is an `others` present for it to have applied to. `str.cat(others=...)` is refused with `UnsupportedError` naming alignment as the missing piece, because pandas aligns the two columns on their labels before it concatenates anything, so a row of `others` meets the row carrying the same label and not the one sitting in the same position, and handing pandas an `others` one row shorter does not raise but matches what it can and fills the rest with missing rows. Concatenating by position instead would be quietly right for every caller whose two columns happen to share an index and quietly wrong for everybody else with no error to say which one they were, which is the worst of the three available options, and refusing it by name puts it on the board as a gap rather than as a disagreement. Document [71](https://github.com/tamnd/firepanda/blob/main/docs/specs/71-an-answer-that-is-narrower-than-a-column.md) in nine sections ([firepanda #790](https://github.com/tamnd/firepanda/pull/790)), five arms and no new registry entry ([compat #166](https://github.com/tamnd/firepanda-compat/pull/166)). The corpus check document 34's habit calls for was run before anything was armed and found the board weaker than it looked for the sixth slice running, and this time the weak case was accusing itself: `strings/cat-scalar` sat on the three text frames that hold no nulls while its own note said it was about what a null does to the answer, which is the one thing those frames cannot show, so an implementation that left a separator standing where a row was dropped would have passed it the moment it was armed. It keeps its frames and its note now says what it actually scores, and `strings/cat-drops-null` runs the same call on `strings_null_heavy`, which is two thirds null with an empty string sitting beside the nulls and therefore separates the dropped row from the empty kept one, a pair nothing on the board told apart before. `strings/cat-na-rep-empty` is the third new case and exists so that the empty stand in and the absent one cannot read as the same request, and `strings/cat-default` is the bare call at L2 because it names no parameter. Nine new scores, all passing, no new divergences and no new failures, L2 178 to 179, L1 483 to 484, L0 487 to 488, L3 unchanged at 102 of 1125 because `cat` cannot reach it while `others` is unimplemented, and the ratchet says nothing went backwards. Eleven `str` names are left: `wrap`, `normalize` and `get_dummies` need no new column type and are the next reachable, though `get_dummies` answers a frame whose width comes out of the data and `normalize` wants the Unicode normalization tables, four want the regular expression engine which is now by some distance the largest blocker on this accessor, `join` and the default form of `split` and `rsplit` want a list column, and `encode` and `decode` want a binary one
- [x] `str.get_dummies`, the third answer shape on this accessor and the first whose width and column names both come out of the data rather than out of the arguments, splitting every row at a separator and answering one int64 column per distinct piece in byte order, keeping the empty string as a real column label because an empty row and a separator at either end both produce it, answering a membership rather than a count so that a token seen twice is still a one, reading a missing row as a row of zeros which is the one place on this accessor a missing row does not stay missing, writing `dtype=bool` and refusing the rest by name, and answering a frame of no columns for a column of only missing rows where pandas' own frame constructor raises `ValueError: Empty data passed with indices specified.`, documented in `docs/specs/72-an-answer-whose-width-is-in-the-data.md`
- [x] `str.normalize`, the one name on this accessor whose answers do not come from Arrow, which is the finding rather than a detail of the implementation. The habit of the last five documents was to work out which of pandas' backends answers a name and to copy that one, and document 64 argued at length that where Arrow and Python's standard library disagree, as they do about what a digit is and about what upper case means for thirty nine code points, Arrow is what pandas answers and Arrow is therefore what gets copied. This name is the other way round. pandas defines `_str_normalize` once, in `ObjectStringArrayMixin`, as `unicodedata.normalize` applied a row at a time, nothing overrides it, `ArrowStringArray` does not have its own, and pyarrow has no normalization kernel for anything to override it with, so on every backend pandas has, including the Arrow backed one that is the default in pandas 3, this is CPython's answer. The rule was never that Arrow is the authority for this accessor. The rule is that the authority for a name is whichever library pandas actually calls for it, found by reading, and it happens to be Arrow for most of them, and getting that backwards here would have produced a kernel built on the Unicode tables Arrow ships, which are a different version from the ones CPython ships, with the disagreements landing in exactly the rare characters nobody writes a test for. It is also the only name here whose rule is about sequences rather than about characters, since what a character normalizes to depends on which characters are next to it, which is why the kernel decodes a whole element before it writes anything and why the tests are about rows of three characters rather than about characters. The four forms are two independent choices rather than four names, which is worth stating because it makes the four fall out instead of having to be memorised: canonical equivalence says two spellings are the same text and compatibility equivalence says they are the same content with the formatting discarded and loses information, and the second choice is whether to finish by taking characters apart or by putting them back together, so the kernel takes two flags and the form name is read into them once in the Python layer. `tools/gen_normalize.py` reads CPython's `unicodedata` and writes `firepanda/kernel/normalize_data.mojo`, and two decisions in it are worth the space. Both decomposition tables are fully expanded before they are written, so one lookup answers the final sequence and the kernel has no recursion in it at all, 2061 characters coming to 3406 code points for the canonical table and 5857 coming to 9112 for the compatibility one, and the argument for that is not size, since the difference is small, it is that a shallow depth is still a depth and the version that walks the chain has a question in it about what to do if the chain does not end. And the composition table is derived rather than read: Unicode refuses to put some decompositions back together under three separate rules, the composition exclusion list, singleton decompositions and decompositions beginning with a non starter, all three fiddly and all three a second chance to be wrong about the same thing, so the generator reads none of them and instead takes every character whose canonical decomposition is exactly two, hands the pair to CPython, and keeps it only when CPython gives the character back, which leaves 941 of 1026 candidates and makes the 85 that drop out the three rules without this code knowing what any of them are. The generator checks itself before it writes, reimplementing all four forms out of nothing but the tables it is about to emit and comparing against CPython over every code point plus 100000 generated awkward sequences, 1212064 rows in total, and refusing to write on a single disagreement, so the tables and the algorithm were both validated before any Mojo was written. Two things cost no table. Korean syllables are arithmetic in both directions, a division to take one apart and a multiplication to put one together, so the 11172 of them are absent from the generated file for the same reason they are absent from CPython's, and they are the largest thing the file does not contain since a table holding them would be three times the size of everything else in it. And an element that is entirely ASCII is already in all four forms and is copied straight through, because no ASCII character has a decomposition of either kind, none has a combining class other than zero and no pair of them composes, all of which is checked in the generator rather than asserted in the kernel. An element that is not valid UTF-8 is copied through too, for the same reason the case kernels do it, that the alternative is a decode which trusts a lead byte and reads off the end of a truncated sequence into the next element, because a text column's payload is one buffer with the elements laid end to end. `normalize` answers text, so it goes through the existing `text` door rather than getting one of its own, which is the door rule working in the other direction after three slices of it adding doors: `partition`, `cat` and `get_dummies` each needed one because each is a new answer shape, and this is the eighth name to arrive without needing an eighth slice because the shape of its answer is a shape that already has a door. Both refusals are pandas' own, a `ValueError` for a form it does not know and a `TypeError` for anything that is not a string, and `form="nfc"` is not an alias in pandas or here, and the form is read twice, once in Python and once in the Mojo door, which is not belt and braces but that the Mojo API is a caller too and an error raised from the kernel with no kind on it reaches Python as something other than the `ValueError` pandas raises. Six rows separate a right implementation from a plausible one and each fails differently: two combining marks in the wrong order catch a missing canonical sort, two marks of the same class catch a sort that is not stable, a letter with a mark of class 202 and a mark of class 230 catch a missing blocking rule so that without it the answer would depend on which mark was typed first which is exactly what normalizing exists to stop it depending on, the angstrom sign is a singleton that composes back to the ordinary letter rather than to the sign it started as, Devanagari qa is on the exclusion list and comes apart under all four forms with nothing putting it back, and the long s with a dot above followed by a dot below, which is UAX 15's own example, keeps its own dot under NFC while under NFKC the long s has already become an ordinary s so both dots land on it and the answer is a character the NFC answer has no route to, which is the row that catches an implementation treating the K forms as a decomposition difference and nothing else. The first four of those pass in an implementation that only asks whether composing after decomposing gives the input back. Document [75](https://github.com/tamnd/firepanda/blob/main/docs/specs/75-the-one-name-answered-by-the-standard-library.md) in eleven sections, 21 Mojo tests and 22 Python ones ([firepanda #797](https://github.com/tamnd/firepanda/pull/797)), three new cases and four new arms ([compat #168](https://github.com/tamnd/firepanda-compat/pull/168)). The corpus check document 34's habit calls for was run before anything was armed and for the eighth slice running it changed what got written, though this time it decided frames rather than exposing a weak case: the unicode frame is the only one where all four forms differ from each other and none is the identity, the folding frame gives four distinct answers with NFC as the identity and separates them on entirely different rows, the pattern frame gives only two, and the ASCII and null heavy frames are the identity under all four, which is why the two K forms take all four text frames, three of them scoring the paths the kernel skips the work on, and the folding case takes the one frame that reaches rows the unicode frame cannot. The three cases already on the board for this name, `strings/normalize-nfc`, `strings/normalize-nfd` and `strings/normalize-len`, were left exactly as they are including their frames, after collection refused a duplicate id written without checking, because ids are stable forever and the frames a case was declared against are part of what it means, so widening the scoring is a new case rather than an edit to an old one. The one note written before it was measured was wrong: the first draft of the folding case claimed the frame's Kelvin sign and long s are the compatibility differences and that none of them produces a combining mark, and measuring it showed no Kelvin sign in the frame at all, the moving rows being the long s, the fi ligature, the micro sign, the dotted capital I and three digraphs, and the dotted capital I does produce one. All twelve runs pass. Board L3 103 to 104 of 1125 with the strings section 16 to 17 of 58 callables, 27.6 percent to 29.3 percent, no new divergences with the registry steady at 30, and the ratchet says nothing went backwards. The only difference either test suite finds against pandas is that a missing row reads out as `None` here and as a float nan there, which is document 63 and is the same on every name in the accessor, and there is a test asserting that is the only difference so the helper flattening it is not quietly hiding anything else. Ten `str` names are left and the piles have not moved: `wrap` needs no new column type and is eleven parameters delegating to `textwrap.TextWrapper`, four want the regular expression engine which is by a long way the largest blocker on this accessor, `join` and the default form of `split` and `rsplit` want a list column, and `encode` and `decode` want a binary one
- [x] The parser that decides which of pandas' two regular expression engines answers a call, which is the front end of #158 rather than a piece of it, and the first thing here that changes no number on the board on purpose. Document 73 measured that every pattern method hands its pattern to Python's own `re._parser` and routes to Python's `re` when the tokens hold a lookaround or a backreference and to Arrow's RE2 otherwise, so the grammar that decides is Python's even though the engine that answers almost everything is RE2. The consequence is the thing worth stating twice: a pattern Python cannot read is not an error in pandas, it is a pattern Arrow gets, so `\p{L}` works there today for the reason that it failed to parse, and every refusal in `firepanda/kernel/regex/parse.mojo` is therefore a routing decision rather than an implementation detail. The nodes live in one flat arena of six integers each, because a node owning a list of children cannot be moved out of the list holding it without copying the subtree underneath and the parser does exactly that move every time a quantifier takes the item in front of it, and because a program counter is an index so the compiler behind this wants indices anyway. Two upstream mistakes are copied on purpose and both are now measured rather than described: the walk enters two of the seven node kinds that can hold another node, so `(?=a)` is answered by Python and `(?=a)?` goes to RE2 and raises an Arrow error naming a library the caller did not call, and `(?!)` collapses to a node that never matches so it routes to the engine that cannot run it while `(?=)` on the same line keeps its node and is answered. The case for reproducing both is that a library which answers where pandas raises has a divergence that only shows up when somebody moves a program back the other way, and it shows up then as a wrong answer instead of an error. `tests/differential/regex.mojo` builds thirty thousand patterns out of the grammar with three quarters of the atoms malformed on purpose and asks both questions of both front ends against a live pandas, calling pandas' own `_has_unsupported_regex` rather than a copy of it, and its first run disagreed on eighty nine patterns out of two thousand: what a quantifier repeats when a comment is in the way, so `a(?#c)+` is `a+`, that a backslash inside a comment hides the bracket after it so `(?#\)` never closes, the four part rule deciding whether a backslash and some digits are an octal number or a backreference, that a backreference is checked where it is written and a conditional at the end of the parse so `\1(a)` is refused and `(?(1)a)(b)` is read, that a global flag group has to be first and an empty alternative counts as something in front of it, that flags can only be turned off in the scoped form so `(?-i)` is a parse error, and that braces which cannot be read as a count are characters. Not one of those seven would have been in a hand written test file, which is the argument for building the corpus before the tests rather than after them. It now runs ten thousand in ten thousand on three seeds with a ceiling of zero, and two kinds of pattern are counted and held out rather than compared: `(?a)(?u)`, where Python raises a `ValueError` that pandas does not catch so the whole `str.contains` call dies and there is no routing decision to agree with, and `\N{NAME}`, where the braces are read and the name is not because there is no Unicode name table yet, with the parse saying so through `approximate` so the matching engine can refuse rather than leaving a pattern that quietly matches the replacement character. Document [76](https://github.com/tamnd/firepanda/blob/main/docs/specs/76-two-engines-and-the-parser-that-picks-one.md) is new at twelve sections ([firepanda #802](https://github.com/tamnd/firepanda/pull/802)), 60 tests beside the differential, and the board does not move because nothing on the accessor is wired to any of it yet, with the registry steady at 30. What is left of #158 is both matching engines, linear time on both sides, two class tables and two end anchors so the two diverge in exactly the four places document 73 measured and nowhere else
- [x] The compiler and the matching engine for the RE2 side, which is the piece of #158 between the parser and anything a caller can reach, and the second slice in a row that moves no number on the board. `firepanda/kernel/regex/program.mojo` turns a parsed tree into a list of nine instructions and `firepanda/kernel/regex/pike.mojo` runs them, in Thompson's program shape and Pike's machine, so every place the pattern could be after reading the same number of characters is held at once and the text is read once. `(a+)+b` against sixty letters is sixty steps, which is not a performance preference but the difference between an engine a caller can hand a pattern to and a denial of service with a friendly API in front of it. Every class in a program lives in one range table laid end to end, so a compiled pattern is two allocations however many classes it holds, and membership is a binary search because a Unicode class table has hundreds of ranges and the classes people write have two. The design decision worth copying is that a refusal carries a flag saying whose it is: either RE2 refuses the pattern too, so refusing here is agreement and the caller gets the same Arrow error pandas gives today, or firepanda cannot answer it yet, so it is a gap with a name. The first harness held out patterns by reading the pattern text, which is wrong twice over, because a pattern where `(?i)` failed to parse is an ordinary RE2 pattern being quietly skipped and because the rule sits where nothing keeps it in step with the compiler, and moving the bit into the compiler fixed both. The refusal is a separate walk over the whole tree rather than something noticed while instructions are written, because `(?Pa)(?P=n){0}` holds a backreference and compiles to no instructions at all. Seven more things RE2 refuses were measured rather than read about, and two of them look like nothing: `\漢` is an ordinary way to write a character to Python and an error to RE2, which recognises no escape outside ASCII, and `[\1]` is the character with code one to Python and an error to RE2, which will not read a nonzero octal escape shorter than two digits because that is how it tells one from a backreference it does not have. The last is a budget rather than a construct: RE2 will not repeat anything more than a thousand times and the limit divides the whole way down, so `(a{11}){91}` is refused for asking for 1001 copies written as two numbers neither of which is over the limit, while `(a*){1000}` is fine and `(a{1000,}){2}` is not because an unbounded repeat spends its lower bound. I had modelled that as a memory budget for twenty minutes and measured it in four. `pixi run differential-regex-match` is the check, thirty thousand generated patterns against sixteen texts through this engine and through pandas' own `str.contains`, comparing the refusal and all sixteen answers, with the texts chosen so every measured difference has something to bite on, and it is ten thousand in ten thousand across five seeds against a ceiling of zero. Captures were left out on purpose with the forty lines written in my head, because the oracle for a boolean answer is `str.contains` and the oracle for a capture has to agree about which of several equal length alternatives a group ends up holding, and that is not measured yet. Document [77](https://github.com/tamnd/firepanda/blob/main/docs/specs/77-the-machine-that-runs-a-pattern-once.md) is new at nine sections ([firepanda #805](https://github.com/tamnd/firepanda/pull/805)), 39 hand written tests beside the differential, the board does not move and the registry stays at 30
- [x] `str.contains`, `str.match` and `str.fullmatch` wired to the engine, which is the first slice of #158 that changes an answer, and the one I had filed as the small one because a binding is where nothing interesting happens. That reading had the component boundary in the wrong place. `match` and `fullmatch` are not modes of any engine, they are `contains` with the pattern rewritten, the rewrite is written by hand in Python inside pandas rather than anywhere near Arrow, and copying it exactly is the whole job. `str.match("a|b")` asks whether a row starts with either letter and not whether it starts with `a` or holds a `b` anywhere, because the rewrite puts the alternation in a group before it puts the anchor on, which is one bracket and a different column, so `tests/test_regex_method.mojo` asserts the rewritten pattern as text rather than only the answer: a rewrite wrong in the same way for every pattern would agree with pandas whenever the engine agreed with RE2, and one that quietly refused a family would agree with pandas on everything it did not refuse. The order upstream does three things in is the part that costs patterns if it is got wrong, and it is route on the pattern as written, then rewrite a trailing `\Z` into RE2's `\z`, then anchor. Routing on the rewrite sends `(?i)(?=a)` to an engine that has never heard of a lookahead, and rewriting `\Z` afterwards is too late because anchoring puts a bracket after it and RE2 refuses a `\Z` that is not last, which was nine patterns per sweep when the differential caught it. A pattern the grammar cannot read is refused over what the caller wrote rather than over the rewrite, since the rewrite adds brackets and brackets can rescue a pattern that was not one, so `)a` is a syntax error where `^()a)` is perfectly good, and that was one pattern in thirty thousand. One thing is not copied and it is forced: pandas hands its rewrite straight to Arrow while this library parses its own rewrite first, and Python's grammar wants a global flag group first in the pattern and refuses one anywhere else, so the group is hoisted to the front. Moving it changes what it covers, and what it now covers that it did not before is exactly the two anchors the rewrite adds, which is harmless for six flag letters and not for `m`: pandas answers `fullmatch("(?m)")` as `[False, False, True, False]` over `['a', 'a\n', '', 'b\nc']` because upstream's added anchors sit outside the group the flag is in and are therefore the ends of the row, and my first hoist answered True for the first row. A hoisted rewrite now writes its anchors as `\A` and `\z`, which are the same two positions with no flag able to touch them, and leaves any caret the caller wrote where it was, both unconditionally rather than only for `m`, because a hoist that is sound for six letters out of seven is one somebody has to keep rechecking. #805's flag saying whose a refusal is now reaches a caller as two different exceptions: a pattern RE2 refuses is a `ValueError`, which is what pandas raises for it out of Arrow, and a pattern this library has not learned yet is a `NotImplementedError`, because somebody who wraps `except ValueError` around a pattern they know to be good should not be told they wrote a bad one. `case=False` with a metacharacter is the third case and is a gap, while a literal pattern with `case=False` keeps the folding path it already had. The differential now runs three sweeps over the same corpus, one per method, and that decision is the one I would repeat without thinking: the `contains` sweep found nothing because it was already green, and the other two found all three defects above, none of which I would have found by writing test cases, because in each one the pattern anybody would think to write is a pattern that works. At thirty thousand cases it is 30052 patterns, 7668 compared under `contains` and 7664 under each of the other two, agreement on every text of every one and zero disagreements in each sweep. An hour of that run went on a disagreement that was not one, where the binary was being run straight out of `build/differential/` instead of through its pixi task, which resolves a different pandas, and the oracle was answering with the previous method's answers, which is a way of being wrong that only became visible once the harness asked three questions in one run. `count` and `replace` did not move, because both need to know where a match ends so they can start looking for the next one and the machine answers whether there is a match rather than where it is, which is the same missing instruction captures need, so the three come together. Document [78](https://github.com/tamnd/firepanda/blob/main/docs/specs/78-three-questions-that-are-one-question.md) is new at twelve sections ([firepanda #810](https://github.com/tamnd/firepanda/pull/810)), 18 tests on the rewrite, 5 on the column kernel against its scalar twin across a morsel boundary and 11 on the wiring against pandas. The board does not move yet and the registry stays at 30, because the compat cases for these three names were written truthfully to describe a `contains` with no engine behind it and rewriting them is its own piece of work on the compat side, so the board now reports these three as worse than they are, which is the better direction to be wrong in and is still wrong
- [x] `str.count` wired to the engine, which is the fourth name of #158 and the first one whose answer is a number rather than a bit, and which I had filed alongside `replace` and captures as one slice because all three need to know where a match ended. That bundle was wrong and measuring it is what took the slice apart. Counting is not asking whether four times. It is a loop, the loop belongs to Arrow rather than to either regular expression engine, and it follows three rules that nobody has written down anywhere: the rest of the row becomes the text after a match, so an anchor is judged against what is left and `str.count("^a")` on `aaa` is three where Python's `re` says one; the cursor moves in bytes and not in characters, so a pattern that can match nothing counts the bytes of a row and one more and a two character Arabic word written in four bytes answers five; and the cursor moves to where the match ended unless the match ended exactly where the cursor already was, so a zero width match found ahead of the cursor is counted where it was found and again from there, which is why `str.count("\b")` on a word with a space on each side of it is two and `str.count("$")` is two for every row there is. The third rule is the one that cost the day. Two simpler rules are easy to reach for, each of them fits the large majority of about sixty measured numbers, and each fails on a pattern people actually write: stepping past an empty match undercounts `\b`, and stepping one byte after every empty match overcounts it, while `\Aa` separates the second from the truth in the other direction. The discipline that found the right one is the discipline this milestone has been running on since document 34, which is measure first and write the rule afterwards, and the one place I relaxed it is the one place it failed: every number in `tests/test_regex_count.mojo` was read off pandas 3.0.5 except one, I derived `count("a|\b")` on four letters as five from the rules I had just written, the engine said four, and pandas says four, so the engine was right and the only number I reasoned my way to was the only wrong one in the file. The engine work underneath this is nine lines. `find` is `matches` with the position of the match recorded, and the Pike machine already ordered its thread list by priority for reasons nothing had needed yet, which is exactly leftmost first, so `a|aa` ends after one character and `aa|a` after two and the difference is invisible in every boolean answer and visible in every count. What is not here is `replace`, and that is now a measurement rather than a guess: `pc.replace_substring_regex` is RE2's `GlobalReplace` and does not share this scan at all, so on the same row `\b` replaces twice where the count says two but `\A` replaces once at the front and `$` once at the end, and `replace` therefore needs its own slice with its own measurements. The findings worth filing upstream are that `count_substring_regex` and `replace_substring_regex` disagree about what the same pattern means in the same library, since one re-slices the text after each match and the other does not, and that `str.count` routes on `_has_unsupported_regex` alone where the three predicates beside it also check `_is_re_pattern_with_flags`. Document [79](https://github.com/tamnd/firepanda/blob/main/docs/specs/79-counting-is-not-asking-four-times.md) is new at ten sections ([firepanda #813](https://github.com/tamnd/firepanda/pull/813)), with 9 Mojo tests holding the three rules one at a time, 11 Python tests on the wiring against pandas, and a fourth differential comparing 7668 patterns over 16 texts, which is 122688 counts, with zero disagreements. On the board `strings/count-regex` gets its arm and three new cases go on beside it, one per rule, chosen so that getting the rule wrong is a wrong number rather than a refusal ([compat #170](https://github.com/tamnd/firepanda-compat/pull/170)): the ascii frame's rows answer their own lengths under an anchored pattern where `re` answers one for all of them, the pattern frame's row of five hundred letters answers five hundred boundaries so the third rule being wrong is off by hundreds rather than by one, and the unicode frame counts bytes under a pattern that can match nothing, which is the other half of the `strings/count-empty` case that asks the literal path the same question. All four pass, the ratchet says nothing went backwards, the registry stays at 30 and L3 is 104 of 1125 with the strings section at 17 of 58 and parameter coverage 55.2 percent. Eight `str` names are left and the piles have not changed shape: `wrap` on its own, four behind the engine, and three behind a column of lists
- [x] `flags` on `str.contains` and `str.fullmatch`, answered out of Python's engine rather than Arrow's, which is the mask half of the scan #158 was waiting on and leaves the counting and replacing halves ahead. The routing could not ride in the flag bits: a flag argument means what the letters mean and it also means the call has moved engines, and `case=False` arrives at the compiler as exactly the same ignore case bit while having to stay on RE2. So the mask door takes a number beside its two words, the letters ride in the number and the route is the number being nonzero, because the door rule that a variant is a new word does not scale to seven letters in any combination. Anchoring moved with the engine, since pandas stops rewriting the pattern the moment it stops talking to Arrow and answers with `regex.fullmatch` instead, so an argued call is pinned with `\A` and `\z` rather than with a `^` and a `$` that move under multiline and sit in front of a trailing newline. The differential then found a fifth difference between the two engines that no amount of reading would have produced: Python's `\B` fails on an empty subject, which is a special case written into CPython in 3.12 rather than a consequence of any rule about word characters, and RE2 has none, so `str.contains(r"\B")` on an empty row answers True and the same call with a flag beside it answers False. Four of the seven letters go through, `re.LOCALE` is a `ValueError` on both sides because Python turns it down on text, and `re.VERBOSE` and `re.ASCII` are gaps that say so. A flag beside `regex=False` turned out not to be the no-op the observations list had it down as, since it is spent on a route that upper cases both sides and disagrees with Arrow about a sharp s and a Kelvin sign the moment a `case` is beside it. 2322 comparisons against live pandas 3.0.5 with no differences and nothing skipped, `differential-regex-match` 10000 in ten thousand with 0 disagreements. Document 85 ([#840](https://github.com/tamnd/firepanda/pull/840))

### Exit criteria

The previous criteria were "every `(M6)` checkbox is ticked with a runnable example", "the differential test passes", and "db-benchmark green at both sizes". Two of those three are not measurable as written, so they are restated as numbers.

- [ ] The compat scoreboard reports L3 at or above the per workstream target for all fifteen issues above
- [ ] Parameter coverage at or above 80 percent for every workstream, so that the L3 numbers mean something. An untouched parameter is not a pass and it is not a failure, it is a hole in the measurement
- [ ] Zero failures. Unimplemented is a schedule and a failure is a bug, and a milestone does not close over a bug
- [ ] The pandas against pandas oracle run is perfect on the same commit, because a harness that disagrees with itself has no business publishing a number about anything else
- [ ] Every divergence found during the milestone is registered with a reason a user would accept, and none of them is `kind = "pending"` at close
- [ ] The operation cost matrix from [`09-resources.md`](https://github.com/tamnd/firepanda-compat/blob/main/docs/specs/09-resources.md) exists for every workstream, with the losses published
- [ ] The full db-benchmark suite runs green at 0.5 GB and 5 GB, **and the results are published whatever they say**

### Three notes

**The string namespace ships whole.** A partial implementation reads as a toy. Half of `.str` is worse than none of it, because a user discovers the gap after they have already ported their program.

**There is no Mojo regex library to start from.** RE2 is the largest single item in this milestone and it is scoped as its own piece of work in #158 rather than absorbed into strings.

**A checkbox above is closed by a number, not by an opinion.** The suite that produces the numbers is [firepanda-compat](https://github.com/tamnd/firepanda-compat), it is built alongside this milestone rather than after it, and it has no skip outcome. A case that has never run is not a pass.

### Depends on

M4.

Start with docs/specs/08-m6.md and the linked child issue for a specific M6 workstream, then run `pixi run conformance` to see its current rate. Done means the workstream reaches its stated L3 target and the CI ratchet accepts it; the overall tracking issue is complete when all unchecked scope items meet their targets.

Written by the indexing model from the issue text.