langgenius / langgenius/dify

Notion page extraction drops mention and equation rich-text segments

Open
#42,170 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

## Description

`NotionExtractor` preserves all Notion rich-text segments in database properties and table cells, but the normal page/block extraction paths still keep only segments that contain a `text` object.

Notion rich text can also contain `mention` and `equation` objects. Those segments expose their rendered value through `plain_text`, so they are currently skipped when a Notion page is imported into a knowledge base.

This affects both:

- `_get_notion_block_data()` for top-level page blocks;
- `_read_block()` for recursively-read child blocks.

## Reproduction

On current `main`, a paragraph payload containing mixed rich text such as:

```python
[
{"type": "text", "plain_text": "See ", "text": {"content": "See "}},
{"type": "mention", "plain_text": "Project Alpha", "mention": {"type": "page", "page": {"id": "page-id"}}},
{"type": "equation", "plain_text": "E = mc^2", "equation": {"expression": "E = mc^2"}},
]
```

is processed by the current loop as only:

```text
See
```

The mention and equation text are omitted because the implementation checks `if "text" in rich_text` and then reads only `rich_text["text"]["content"]`.

## Expected behavior

All rendered rich-text segments should be preserved in extracted page content, using each segment's `plain_text` value (with the existing `text.content` shape retained only as a fallback if needed).

For the payload above, the extracted text should contain:

```text
See Project AlphaE = mc^2
```

## Upstream precedent

This is consistent with the recently merged table-cell fix in #41993. During review, `plain_text` was explicitly preferred because Notion rich text can contain `text`, `mention`, and `equation` segments, and the merged `_get_cell_text()` helper now follows that rule.

The recently merged database-property fix in #42050 also joins all `plain_text` segments instead of taking only a subset.

The two ordinary page/block paths appear to be the remaining sibling omission in the same extractor.

## Proposed scope

Keep the fix narrow:

1. preserve `plain_text` for every rich-text segment in `_get_notion_block_data()`;
2. do the same in `_read_block()`;
3. add focused regression coverage for top-level and nested blocks containing text + mention + equation segments;
4. leave database-property and table-cell behavior unchanged.

I have the scoped change ready and will link the PR here.

Contributor guide

Open the contributing guide

Research direction

Start with the rich-text handling in _get_notion_block_data() and _read_block(), comparing it with the recently merged _get_cell_text() behavior from #41993. Add focused regression coverage for top-level and nested blocks containing text, mention, and equation segments; done means every segment's plain_text is preserved while the existing text.content fallback remains supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.