huggingface / huggingface/transformers.js

How to use xenova/transformers in VSCode Extension

Open
#317 7 comments 2 reactions 0 assignees View on GitHub
question
Dominant language
JavaScript
Stars
16.3k
Forks
1.2k
Avg merge
6d 2h
Merged PRs (30d)
6

Description

Hey guys! I am trying to use xenova/transformers in CodeStory, we roll a vscode extension as well and I am hitting issues with trying to get the import working, here's every flavor of importing the library which I have tried to date.

```
const TransformersApi = Function('return import("@xenova/transformers")')();
const { pipeline, env } = await TransformersApi;
```

```
const { pipeline, env } = await import('@xenova/transformers')
```

```
const TransformersApi = require('@xenova/transformers');
const { pipeline, env } = await TransformersApi;
```

I think the crux of the issue is the node environment which VSCode uses which does not allow any of these to work, and I keep getting the deaded:

```
Error [ERR_REQUIRE_ESM]: require() of ES Module /Applications/Aide.app/Contents/Resources/app/extensions/codestory/node_modules/@xenova/transformers/src/transformers.js from /Applications/Aide.app/Contents/Resources/app/extensions/codestory/out/llm/embeddings/sentenceTransformers.js not supported.
Instead change the require of transformers.js in /Applications/Aide.app/Contents/Resources/app/extensions/codestory/out/llm/embeddings/sentenceTransformers.js to a dynamic import() which is available in all CommonJS modules.

```

after checking the js code which is generated, it ends up including the require word:
```
__importStar(require('@xenova/transformers'))
```

when I used the first option which was a function I got a very weird error btw:
```
[Extension Host] TypeError: A dynamic import callback was not specified.
at new NodeError (node:internal/errors:399:5)
at importModuleDynamicallyCallback (node:internal/process/esm_loader:39:9)
at eval (eval at (/Applications/Aide.app/Contents/Resources/app/extensions/codestory/out/llm/embeddings/sentenceTransformers.js:46:41), :3:1)

```

this is mostly comping from the node version which VSCode uses itself.

Do you guys have any suggestions on what I can do about this? Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.