OpenFn / OpenFn/lightning

Adaptor docs sometimes pulls down too much data (and so is slow)

Open
#524 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

This issue doesn't technically belong here, it's kind of a cross-repo thing. But it's impact is here and it can be managed more effectively from here I think.

The Issue

When loading documentation for some pre-monorepo adaptors, the client can download far more information than it needs. language-common@1.7.1 for example, downloads hundreds of files (!!), meaning it takes docs a lot longer to generate than it should.

The problem is:

  • adaptor-docs asks kit/describe-package for metadata about a package
  • describe-package loads the package.json manifest for that package, and downloads every d.ts file listed in the files array
  • Some (all?) adaptor package.jsons list files from its node_modules - files which surely shouldn't be listed.

It is probably worth examining the impact a bit more before deciding the severity of this. To be fair I think it's quite a simple fix.

Solutions

Probably the best fix is in kit/describe-package. When fetching a file listing from npm, we simply ignore anything that starts with node_modules.

An alternative is to wait for the doc builder to move out of the client and into the server (see #492). With caching in place the impact of this is greatly reduced. We may consider fixing describe-package but it'll be less critical.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with adaptor-docs and kit/describe-package, then inspect how the package.json files array is used when fetching the language-common@1.7.1 file listing. Confirm the excess downloads and determine whether node_modules entries can be ignored there; done means documentation generation no longer fetches those unnecessary files.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js
Domain
documentation, performance, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.