Automattic / Automattic/cortext

Bulk-write documents on Notion import

Open
#302 0 comments 0 reactions 0 assignees View on GitHub
area: performance type: enhancement
Dominant language
JavaScript
Stars
51
Forks
2
Avg merge
3h 12m
Merged PRs (30d)
8

Description

The Notion importer (#275) creates documents one at a time. Each is a separate insert, plus its meta, plus a term assignment, so a large database means a lot of round-trips.

**Proposed solution:** batch the writes. Insert documents in chunks, prime caches once, and write meta and trait terms in bulk instead of per document.

Contributor guide

Open the contributing guide

Research direction

Start with the Notion importer described in issue #275 and trace its per-document insert, metadata write, and term assignment. Identify the existing cache initialization and database write paths; done means documents are inserted in chunks, caches are primed once, and metadata and trait terms are written in bulk without changing import results.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
backend, database
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.