describe-package: create a simple, all-powerful describe(specifierOrPath) function

describe-package needs quite a lot of love. It's been twisted around a bit and half refactored and never really had the clean high level interface it needs.

What it needs to be able to do is:

• Provide a describe API
• Describe takes a specifier (ie @openfn/language-http@1.0.0) or a path (ie, repo/@openfn/language-http_1.0.0)
• Maybe it takes a repo path as well - because the CLI can tell it where to load other dependencies locally, rather than calling out to unpkg
• Given a specifier, it will call out to unpkg (or the repo?) to resolve the types of the provided module
• If given a path, it;'ll use the path (and maybe the repo)
• Either way the API must:
  • Look for the typings entrypoint
  • Describe any exports
  • Including anything exported from other packages (this is the hard bit)

Lightning and the CLI both need to use this function to describe the interface of a package. Both systems currently implement their own hard-coded pre-load of langauge-common to provide types for common exports.

Spec Work

EDIT: This is a WIP, but it'll replace the above comments

Overview

A quick overview of what describe-package should do:

• Download a package on jsdeliver
• Download a single file from a package on jsdeliver
• For a file or package, generate a propritary JSON representation of the public API
• Generation is based on TypeScript's analysis of the package

The finished rebuilt package should be version bumped to 1.0.0. We can change the name if we think it's appropriate.

API

TODO: clarify the actual API in detail

describe(specifier, options)

Options includes:

• format: json, dts (for Lightning) , markdown (for testing)
• includePrivate: if true, ignore any JSDOC visibility on functions and return everything exported by the adaptor. By default this is false, and describePackage only includes public documentation.
• source: or something This indicates where to load definitions from. By default this is jsdelivr. But why not support npmjs or unpkg (we don't want to do either today, but I'd love to have a hook for this)? Why not support the CLI repo? Why not support an arbitrary folder on disk? In fact, we kinda need to!

A big problem

I want the ability to load type definitions from a local file on disk

But obviously the browser doesn't have access to the file system

So how will local lightning fetch local type definitions? We'll have to set up a proxy. hmm.

Implementation Details

The current implementation has a proprietary abstraction layer over the TS engine to make it a bit simpler to use. This was probably more useful as a learning experience than as an actual interface to the TS compiler. We can keep it and enhance it, or ditch it entirely. We should consider something like ts-morph as a better utility/abstraction layer

The current implementation has a bunch of stuff to build a web worker to do this processing. I don't think we need it - all that stuff can go. It works in the browser just fine now (amazingly), and I'd actually prefer to push this logic out into a web server.

There is some gnarly stuff in there about downloading the type defs for common as well as the main adaptor. I'd like to relax that, or find a more generic solution, because:

• Right now every adaptor extends common. But I would like to start doing more interesting dependencies on adaptors. Really, we should look at the public exports of the function and properly track the types through the dependency graph.
• We also have a thing now where we have this secret collections API. Every job in lightning has access to the collections adaptor, as well as whatever adaptor the user chooses. So should we also add the collections functions? Maybe we need an option to pass multiple specifiers (simulating multiple adaptors in a single step).

A note on history

• Originally, describe-package was a v2 compiler. That got replaced long ago by @openfn/compiler
• We kept some of that functionality but renamed it describe-package and plugged a new, higher level interface into api.ts. This was supposed to be a clean interface which reconciled all the functionality - but I never got around to finishing it :(
• describeDts and fetchFile are the older APIs which I would like to be remove or make private. This uses the describeProject function internally
• describePackage is the newest entry point as part of the refactor.

Dependencies

Who exactly depends on describe-package right now?

• Lightning's adaptor-docs component. This calls describePackage(specifier) to get the JSON representation of the package. It then renders it with react and tailwind. It's cool.
• Lightning's main editor, the monaco implementation, also uses it. Monaco calls fetchDTSListing and fetchFile to download specific files, which get loaded into Monaco's ts environment. The editor component works quite hard to make this work. It would be great to do something like desrcribePackage(adaptor@version, { format: 'dts' }) to get a flat DTS interface to drive this functionality. I am probably happy to mimic any hacks that monaco needs.
• @openfn/cli uses the main describePackage() function to print docs to the console. It's cool that we do this. I don't think anyone uses it. It should be compatible with the new API above (true at the time of writing)
• @openfn/compiler uses the describeDts (and Project) functions to work out what the exports of an adaptor are. A bit like Lightning, it works a bit too hard to do this. Literally all it wants is the name of the top level exports. I would love to refactor this to use describePackage(specifier). Note that it will want to use local module paths, so we need to ensure our new function is flexible

For what it's worth, I also want to build a describe-package service and stick it on a server somewhere for lightning to call out externally. This itself needs more thought (no-one wants lightning to depend on yet another server). I don't think this affects anything in the design but let's keep this in mind.

Demo

I'd like a clean demo / test harness for the update. Mostly so that I can cheaply validate if it's working.

In examples there is a dts-inspector app. This was the original test harness. It probably needs to be removed.

We could take lighting's adaptor-docs component and use that to render the docs. That makes it really simple to see the output. If it's rendered into HTML correctly, we know it's parsed the source correctly.

On the other hand, I don't want to duplicate the code or have a complicated tailwind server.

How about we add a suite of integration tests which calls describe-package on a few real adaptors, then generates some very simple markdown output. Maybe it's even useful to have a toMarkdown() util function in describe-package - we can unit test that locally and then run it in the integration test on real adaptors.

Start with api.ts and the existing describePackage, describeDts, fetchFile, and describeProject entry points to map the current behavior and their consumers. Review Lightning adaptor-docs, the CLI, and compiler usage, then use the examples/dts-inspector harness or integration tests to agree on the API, supported formats and sources, dependency handling, and a demonstrable finished result.