docling-project / docling-project/docling
Feature Request: Semantically Chunk Code Files
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Requested feature
For many RAG applications, document processing is the most common use case. In multi-source RAG setups, code repositories often serve as a key knowledge source.
I'm requesting support for processing code files based on Abstract Syntax Trees (ASTs). Instead of breaking chunks at a character threshold, the system would produce semantic chunks, one function or class per chunk (unless it exceeds a configurable length threshold).
A possible implementation could leverage [tree-sitter](https://tree-sitter.github.io/tree-sitter/). Some existing packages that demonstrate this approach include:
- [chonkie](https://github.com/chonkie-inc/chonkie)
- [chunkhound](https://github.com/chunkhound/chunkhound)
Contributor guide
Assessment
This issue has not been assessed yet.