Docsite search: Test different chunking strategies
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 5
- Forks
- 10
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 17
Description
So this splits the chunk at the first line that goes over the limit?
Is the cutoff chunk now going to be massively out of context?
I don't think we need to solve this today but maybe we should raise out an issue for it.
My instinct is that there are all sorts of heuristics we can apply here: cutting out whole HTML elements (section/detail) for example, or splitting out code blocks, or tip boxouts.
Originally posted by @josephjclark in https://github.com/OpenFn/apollo/pull/176#discussion_r1987804566
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the chunking discussion in pull request #176, especially the referenced comment, and trace how the docsite search currently splits content. Compare the proposed HTML-aware strategies and define how their search context should be evaluated; done requires an agreed strategy and evidence from those comparisons.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- html
- Domain
- documentation, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100