apache / apache/pulsar-client-node

Problems with using a bundler (esbuild, webpack, etc)

未关闭
#397 1 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C++
星标
164
派生
98
PR 合并指标
30 天内没有已合并 PR

描述

We haven't been able to get pulsar-client to work with a bundler like webpack or esbuild. The `pulsar.node` file is not picked up by the bundler and additional files in the library (.html, .cs) end up breaking the build. Even if the bundler is configured to ignore these html/cs files, it fails to export `pulsar.node` file from node_modules into the build directory.

An alternative path is to get webpack/esbuild to ignore `pulsar-client` completely and to manually copy over `pulsar.node` from node_modules. This ends up causing problems because [pulsar-binding.js](https://github.com/apache/pulsar-client-node/blob/master/src/pulsar-binding.js) explicitly looks for the package.json file and for mapbox/node-pre-gyp:

```
const binary = require('@mapbox/node-pre-gyp');

const bindingPath = binary.find(path.resolve(path.join(__dirname, '../package.json')));
```

So it seems like pulsar-client would need to be architected differently to work with webpack/esbuild.

We did finally find a workaround to the problem. Sharing it here for other people who run into this too: we use esbuild to build and package up our node.js app but we add an additional step that creates a simple package.json file with pulsar-client in the parent directory and then run `npm install`. This way we can continue using esbuild to package up our node app but externally inject pulsar-client as a dependency.

Here's an example build.js for esbuild:

```
const esbuild = require("esbuild");
const { copy } = require("esbuild-plugin-copy");
const fs = require("fs-extra");
const path = require("path");
const { execSync } = require("child_process");

// Ensure that native modules, like node-canvas, are bundled correctly
// This however does not work for pulsar-client, see the then() block below
const nativeNodeModulesPlugin = {
name: "native-node-modules",
setup(build) {
build.onResolve({ filter: /\.node$/, namespace: "file" }, (args) => ({
path: require.resolve(args.path, { paths: [args.resolveDir] }),
namespace: "node-file",
}));

build.onLoad({ filter: /.*/, namespace: "node-file" }, (args) => ({
contents: `
import path from ${JSON.stringify(args.path)}
try { module.exports = require(path) }
catch {}
`,
}));

build.onResolve({ filter: /\.node$/, namespace: "node-file" }, (args) => ({
path: args.path,
namespace: "file",
}));

const opts = build.initialOptions;
opts.loader = opts.loader || {};
opts.loader[".node"] = "file";
},
};

esbuild
.build({
entryPoints: ["src/index.ts"],
plugins: [nativeNodeModulesPlugin],
bundle: true,
platform: "node",
target: "node18",
outdir: "build/src",
tsconfig: "tsconfig.json",
sourcemap: true,
// Ignore pulsar-client and related dependencies. These will be installed in the build directory later.
external: ["*.html", "mock-aws-s3", "aws-sdk", "nock", "pulsar-client"],
})
.then(() => {
// pulsar-client does not work when bundled so we install it in the build
// directory as a dependency.

// Get the version of pulsar-client that we are using
const pulsarVersion = require("pulsar-client/package.json").version;
// Create a package.json file in the build directory with pulsar-client as a dependency
const packageJson = {
name: "external-deps",
version: "1.0.0",
dependencies: {
"pulsar-client": `^${pulsarVersion}`,
},
};
// Write the package.json file to the build directory and install the dependencies
fs.writeFileSync(path.join("build", "package.json"), JSON.stringify(packageJson, null, 2));
execSync("cd build && npm install --no-package-lock");
})
.catch(() => process.exit(1));

```

We then launch our app: `cd build && node ./src/index.js`

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先使用 webpack 或 esbuild 重现 bundling 失败,重点关注 src/pulsar-binding.js 以及 @mapbox/node-pre-gyp 对 ../package.json 的查找。跟踪来自 node_modules 的 pulsar.node 和其他 HTML/CS 文件是如何处理的。当有一条经过文档记录或测试的 bundler 路径能够在不使用外部安装 workaround 的情况下打包并加载 pulsar-client 时,即可视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
javascript, nodejs, webpack
领域
build-system
Issue 类型
缺陷
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。