docling-project / docling-project/docling

Feature Request: Semantically Chunk Code Files

Open
#2,351 2 comments 0 reactions 0 assignees View on GitHub
enhancement triage/close-stale
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Requested feature

For many RAG applications, document processing is the most common use case. In multi-source RAG setups, code repositories often serve as a key knowledge source.

I'm requesting support for processing code files based on Abstract Syntax Trees (ASTs). Instead of breaking chunks at a character threshold, the system would produce semantic chunks, one function or class per chunk (unless it exceeds a configurable length threshold).

A possible implementation could leverage [tree-sitter](https://tree-sitter.github.io/tree-sitter/). Some existing packages that demonstrate this approach include:
- [chonkie](https://github.com/chonkie-inc/chonkie)
- [chunkhound](https://github.com/chunkhound/chunkhound)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.