[Bug]: DefaultTableExtraction drops <th> row headers and ignores rowspan, producing misaligned result.tables

crawl4ai version

0.9.3

Expected Behavior

result.tables[i]["rows"] should match the logical grid a browser renders:

• Row-header cells (<th scope="row"> inside <tbody>) are kept as the first column.
• A rowspan cell's value is repeated in every row it covers.
• cells keep the current "repeat the value" behaviour, but the row must stay aligned with the header (no left shift, no phantom trailing "").

Current Behavior

DefaultTableExtraction.extract_table_data only iterates .//td for body rows, so every <th> inside <tbody> is dropped: the remaining cells shift left and the last column is padded with "". rowspan is not handled at all. For manual documentation tables whose first column is a product / parameter name in <th>, the whole key column disappears from result.tables.

This is the follow-up to #2007: since v0.9.1 rowspan/colspan survive in cleaned_html, but the extractor does not use them.

Is this reproducible?

Yes

Inputs Causing the Bug

Any table with (a) `<th scope="row">` cells in the body, or (b) a `rowspan` > 1 cell. Minimal HTML is in the steps below.

Steps to Reproduce

1. Run the script below (no browser, no crawl).
2. Compare the printed rows with the expected rows.

Code snippets

from lxml import html as lhtml
from crawl4ai import DefaultTableExtraction

HTML = """<table>
<tr><th></th><th>Feature A</th><th>Feature B</th></tr>
<tr><th scope="row">Item 1</th><td>yes</td><td>yes</td></tr>
<tr><th scope="row">Item 2</th><td>no</td><td>yes</td></tr>
</table>
<table>
<thead><tr><th>Group</th><th>Option X</th><th>Option Y</th></tr></thead>
<tbody>
<tr><td rowspan="2">Group 1</td><td>value x</td><td>value y</td></tr>
<tr><td colspan="2">note that applies to X and Y</td></tr>
</tbody></table>"""

root = lhtml.fromstring(HTML)
for table in DefaultTableExtraction().extract_tables(root):
    print(table["headers"], table["rows"])

OS

windows 11

Python version

3.12

Browser

N/A

Browser version

N/A

Error logs & Screenshots (if applicable)

Actual:
['', 'Feature A', 'Feature B'] [['yes', 'yes', ''], ['no', 'yes', '']]
['Group', 'Option X', 'Option Y'] [['Group 1', 'value x', 'value y'], ['note that applies to X and Y', 'note that applies to X and Y', '']]

Expected:
['', 'Feature A', 'Feature B'] [['Item 1', 'yes', 'yes'], ['Item 2', 'no', 'yes']]
['Group', 'Option X', 'Option Y'] [['Group 1', 'value x', 'value y'], ['Group 1', 'note that applies to X and Y', 'note that applies to X and Y']]

I'd like to submit a PR: expand rowspan/colspan into a rectangular grid (pending-cell map), collect th|td for body rows, and add unit tests for the cases above. The change is limited to crawl4ai/table_extraction.py plus a small shared grid helper; the scoring logic in is_data_table is untouched.

Start in crawl4ai/table_extraction.py, especially DefaultTableExtraction.extract_table_data, and run the provided lxml reproduction script. Add coverage for body-row th cells plus rowspan and colspan cases, keeping rows aligned with the headers and repeating values across covered cells. Done means the expected rows in the issue are produced and the new unit tests pass.