indentlabs / indentlabs/notebook

Come up with a plan to address content page scaling issues

Open
#331 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Needs Input
Dominant language
HTML
Stars
410
Forks
78
Avg merge
1d
Merged PRs (30d)
9

Description

All of our content pages exist in their own database tables right now, which is great for lookup speed. However, in order to get all of a user's content in a single query, we're currently doing a massive join of all these tables and we're pretty much at the memory limit for in-memory results. This is limiting us from creating additional page types.

We should figure out a way to address this limitation and remove it. Some ideas:

  • Modify the existing query to use less memory during the query, allowing us to join across more tables at once without hitting resource limits.

  • Figure out a different way to query for all of a user's content at once, so the massive join query is no longer necessary.

  • Migrate the UI to a JS front-end that can load each page type piecemeal (from separate requests) as the data comes available

  • Other solutions?

In any case, we'd ideally like a solution that scales well with increasingly large numbers of content pages over time (both in the number of user-created pages as well as the number of different types of pages). Eventually we'd like to be able to allow users to create their own page types (creating the template from scratch), so being unbounded on N page types in the database (while still retaining fast lookup speeds) would be ideal.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the separate content-page tables and the massive query that combines them for a user. Compare the proposed query, data-model, and UI-loading approaches; done means selecting and documenting a scalable design that avoids the current memory limit while supporting more page types.

Written by the indexing model from the issue text.

Assessment

Domain
backend, databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.