Markdown knowledge import removes literal hashes from headings (C# becomes C)
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
- [x] I have searched for existing issues, including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.
### Dify version
main at 5e51fe2640 (api version 1.17.0)
### Cloud or Self Hosted
Self Hosted (Source). Reproduced at the extractor level; no running deployment was used.
### Steps to reproduce
The built-in Markdown extractor is selected for `.md`, `.markdown`, and `.mdx` knowledge uploads when using the default ETL path. Its heading cleanup removes every `#`, including characters that belong to the title text.
A minimal extractor reproduction, run from `api/` in a configured development environment:
```python
from pathlib import Path
from tempfile import TemporaryDirectory
from core.rag.extractor.markdown_extractor import MarkdownExtractor
with TemporaryDirectory() as directory:
path = Path(directory) / "example.md"
path.write_text("# C#\nLanguage reference.\n\n## Issue #123\nIssue details.\n", encoding="utf-8")
docs = MarkdownExtractor(str(path), encoding="utf-8").extract()
for doc in docs:
print(repr(doc.page_content))
```
### ✔️ Expected Behavior
The nonempty documents retain `C#` and `Issue #123` as their titles. Opening heading markers and optional closing heading markers may still be removed, as they are today.
### ❌ Actual Behavior
The titles become `C` and `Issue 123`. This silently changes the text ingested into the knowledge base, including programming language names and issue references.
The cause is `re.sub(r"#", "", key)` in `MarkdownExtractor.markdown_to_tups`, which removes hashes throughout the title.
This differs from #30940, which requested retaining the Markdown heading markers themselves. This report concerns literal characters in the heading content.
I would like to contribute a small fix that removes only the opening marker and valid optional closing markers, with parameterized regression coverage through `extract()`. Could a maintainer confirm the approach and assign this issue to @gallonyin?
Contributor guide
Research direction
Start in api/core/rag/extractor/markdown_extractor.py at MarkdownExtractor.markdown_to_tups and follow the extract() entry point. Reproduce the example with headings such as C# and Issue #123, then add parameterized regression coverage through extract(). Done means opening and valid optional closing markers are handled while literal hashes in heading text remain.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100