Break files into functions or classes
- Dominant language
- TypeScript
- Stars
- 40
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
Currently, the token limit of the gpt models prevents us from indexing large source files. If we broke the files up into smaller chunks (like classes, or even functions), we could index (nearly?) everything. Ideally we keep the largest chunks possible though so we don't lose semantic information about cross-method or cross-function calling patterns.
This could either be done with some language-specific parsing library... or even better, using some LLM-assisted chunking that can work for (nearly) any kind of source file. ( https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents )
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the repository's current source indexing flow and the chunking requirements described in issue #5, then compare the language-specific and LLM-assisted approaches, including the linked Azure guidance. Done means large source files can be indexed within model token limits while keeping chunks as large as possible without losing cross-function or cross-method context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100