indentlabs / indentlabs/notebook
Come up with a plan to address content page scaling issues
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 410
- Forks
- 78
- Avg merge
- 1d
- Merged PRs (30d)
- 9
Description
All of our content pages exist in their own database tables right now, which is great for lookup speed. However, in order to get all of a user's content in a single query, we're currently doing a massive join of all these tables and we're pretty much at the memory limit for in-memory results. This is limiting us from creating additional page types.
We should figure out a way to address this limitation and remove it. Some ideas:
-
Modify the existing query to use less memory during the query, allowing us to join across more tables at once without hitting resource limits.
-
Figure out a different way to query for all of a user's content at once, so the massive join query is no longer necessary.
-
Migrate the UI to a JS front-end that can load each page type piecemeal (from separate requests) as the data comes available
-
Other solutions?
In any case, we'd ideally like a solution that scales well with increasingly large numbers of content pages over time (both in the number of user-created pages as well as the number of different types of pages). Eventually we'd like to be able to allow users to create their own page types (creating the template from scratch), so being unbounded on N page types in the database (while still retaining fast lookup speeds) would be ideal.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the separate content-page tables and the massive query that combines them for a user. Compare the proposed query, data-model, and UI-loading approaches; done means selecting and documenting a scalable design that avoids the current memory limit while supporting more page types.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100