xorbitsai / xorbitsai/deepdoc-lib
feat: implement document chunking for long PDFs
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
Problem
Long documents currently require excessive GPU memory allocation.
Current workaround
Commit fe8a934 increases GPU memory allocation, which is not scalable.
Proposed solution
Split long PDF documents into smaller chunks before processing instead of allocating more GPU memory.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how long PDF documents are currently processed and review commit fe8a934, which increases GPU memory allocation. Define the chunking boundary before processing and verify that long PDFs can be handled without relying on increased GPU memory allocation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100