langgenius / langgenius/dify

Markdown knowledge import removes literal hashes from headings (C# becomes C)

Open Beginner friendly
#41,907 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
- [x] I have searched for existing issues, including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.

### Dify version

main at 5e51fe2640 (api version 1.17.0)

### Cloud or Self Hosted

Self Hosted (Source). Reproduced at the extractor level; no running deployment was used.

### Steps to reproduce

The built-in Markdown extractor is selected for `.md`, `.markdown`, and `.mdx` knowledge uploads when using the default ETL path. Its heading cleanup removes every `#`, including characters that belong to the title text.

A minimal extractor reproduction, run from `api/` in a configured development environment:

```python
from pathlib import Path
from tempfile import TemporaryDirectory
from core.rag.extractor.markdown_extractor import MarkdownExtractor

with TemporaryDirectory() as directory:
path = Path(directory) / "example.md"
path.write_text("# C#\nLanguage reference.\n\n## Issue #123\nIssue details.\n", encoding="utf-8")
docs = MarkdownExtractor(str(path), encoding="utf-8").extract()
for doc in docs:
print(repr(doc.page_content))
```

### ✔️ Expected Behavior

The nonempty documents retain `C#` and `Issue #123` as their titles. Opening heading markers and optional closing heading markers may still be removed, as they are today.

### ❌ Actual Behavior

The titles become `C` and `Issue 123`. This silently changes the text ingested into the knowledge base, including programming language names and issue references.

The cause is `re.sub(r"#", "", key)` in `MarkdownExtractor.markdown_to_tups`, which removes hashes throughout the title.

This differs from #30940, which requested retaining the Markdown heading markers themselves. This report concerns literal characters in the heading content.

I would like to contribute a small fix that removes only the opening marker and valid optional closing markers, with parameterized regression coverage through `extract()`. Could a maintainer confirm the approach and assign this issue to @gallonyin?

Contributor guide

Open the contributing guide

Research direction

Start in api/core/rag/extractor/markdown_extractor.py at MarkdownExtractor.markdown_to_tups and follow the extract() entry point. Reproduce the example with headings such as C# and Issue #123, then add parameterized regression coverage through extract(). Done means opening and valid optional closing markers are handled while literal hashes in heading text remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.