bjsi / bjsi/incremental-everything
build a SuperMemo-like Web Importer
- Dominant language
- TypeScript
- Stars
- 75
- Forks
- 11
- Avg merge
- 1m
- Merged PRs (30d)
- 11
Description
>What would RichTextNamespace.parseAndInsertHtml be useful for? Would it be able to parse an html site (e.g. https://en.wikipedia.org/wiki/Strait_of_Hormuz) and insert its content in a RemNote-friendly way? I ask this because I am searching for a SuperMemo-like web importer (https://help.supermemo.org/wiki/Web_import), that parses and import the content locally.
Yes, `RichTextNamespace.parseAndInsertHtml` is exactly the right tool for the final step of building a SuperMemo-style web importer, but you will need to do some intermediate work to get the clean result you want.
According to the SDK, `parseAndInsertHtml` takes an HTML string and a target `RemId` (or `PluginRem`), parses that HTML, and inserts it as a tree of Rem. Under the hood, this function uses RemNote's native paste-parsing logic. It understands semantic HTML tags (like `
`, ``, ``, ``, ``) and converts them into a hierarchical Rem structure with rich text formatting applied.
### The Catch: You can't just pass raw website HTML
If you fetch the raw HTML of `https://en.wikipedia.org/wiki/Strait_of_Hormuz` and pass it directly to `parseAndInsertHtml`, RemNote will attempt to parse *everything*. You will end up with hundreds of garbage Rems generated from the site's navigation bars, side panels, footer links, hidden metadata, and script tags. RemNote's parser translates HTML to Rem, but it does not have an "Article Extractor" built-in.
### How to build your SuperMemo Web Importer
To successfully replicate SuperMemo's Web Import feature, your plugin should follow a three-step pipeline:
1. **Fetch the HTML:** Use `fetch()` or a proxy (to bypass CORS, which is often an issue with client-side fetching) to get the raw HTML of the target URL.
2. **Extract the Article (The Missing Link):** Before handing the HTML to RemNote, you must strip away the boilerplate web noise. You can do this by running the HTML through a library like **Mozilla's Readability.js** (the same engine that powers Firefox's Reader View). This will give you a clean HTML string containing only the main article's title, headers, text, and images.
3. **Insert into RemNote:** Pass that cleaned HTML string to `plugin.richText.parseAndInsertHtml(cleanedHtml, targetRem.id)`.
### Alternative Approach: HTML to Markdown
If you find that `parseAndInsertHtml` interprets certain HTML tags in ways you don't like, another popular approach is to convert the cleaned HTML into Markdown (using a library like `Turndown.js`) and then use `plugin.rem.createTreeWithMarkdown(markdown, parentId)`. This gives you a bit more manual control over exactly how the hierarchy is constructed before RemNote processes it.
`, ``, ``) and converts them into a hierarchical Rem structure with rich text formatting applied.
### The Catch: You can't just pass raw website HTML
If you fetch the raw HTML of `https://en.wikipedia.org/wiki/Strait_of_Hormuz` and pass it directly to `parseAndInsertHtml`, RemNote will attempt to parse *everything*. You will end up with hundreds of garbage Rems generated from the site's navigation bars, side panels, footer links, hidden metadata, and script tags. RemNote's parser translates HTML to Rem, but it does not have an "Article Extractor" built-in.
### How to build your SuperMemo Web Importer
To successfully replicate SuperMemo's Web Import feature, your plugin should follow a three-step pipeline:
1. **Fetch the HTML:** Use `fetch()` or a proxy (to bypass CORS, which is often an issue with client-side fetching) to get the raw HTML of the target URL.
2. **Extract the Article (The Missing Link):** Before handing the HTML to RemNote, you must strip away the boilerplate web noise. You can do this by running the HTML through a library like **Mozilla's Readability.js** (the same engine that powers Firefox's Reader View). This will give you a clean HTML string containing only the main article's title, headers, text, and images.
3. **Insert into RemNote:** Pass that cleaned HTML string to `plugin.richText.parseAndInsertHtml(cleanedHtml, targetRem.id)`.
### Alternative Approach: HTML to Markdown
If you find that `parseAndInsertHtml` interprets certain HTML tags in ways you don't like, another popular approach is to convert the cleaned HTML into Markdown (using a library like `Turndown.js`) and then use `plugin.rem.createTreeWithMarkdown(markdown, parentId)`. This gives you a bit more manual control over exactly how the hierarchy is constructed before RemNote processes it.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the SDK entry points mentioned in the issue: plugin.richText.parseAndInsertHtml and plugin.rem.createTreeWithMarkdown. Then investigate fetching HTML, Mozilla Readability.js, and Turndown.js; the work is done when a target page can be cleaned and imported locally into a RemNote-friendly hierarchy without surrounding site boilerplate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- web-dev
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100