llmware-ai / llmware-ai/llmware
Library.add_files(params, max_chunk_size=n) often creates record in db where chunk size vastly exceed n - often representing an entire document page of text
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.8k
- Forks
- 2.9k
- PR merge metrics
- No merged PRs in 30d
Description
Library.add_files(params, max_chunk_size=n) often creates record in db where chunk size vastly exceed n - often representing an entire document page of text
simply as described.
appears to be more associated with the parsing of pdf documents that have entire pages comprised of a scanned image
are these types of record included in embedding? if so, problematic, right?
macos 15.x
llmware v 0.3.8
active_db: sqlite
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the Library.add_files(params, max_chunk_size=n) entry point and reproduce the behavior with a PDF containing scanned-image pages using the active SQLite database. Trace how those pages become database records and whether they are included in embeddings. Done means the observed chunk-size behavior and embedding treatment are either corrected or documented with a focused regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100