Blizzard / Blizzard/node-rdkafka

Segmentation fault and build errors using node16 and OpenSSL

Open
#1,001 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.2k
Forks
403
PR merge metrics
No merged PRs in 30d

Description

OS:
Name: Amazon Linux
Version: 2
ID Like: centos rhel fedora
Architecture: x86_64

Installed packages (yum):
openssl.x86_64
openssl-devel.x86_64
openssl-libs.x86_64
openssl11.x86_64
openssl11-libs.x86_64
openssl11-pkcs11.x86_64
gcc.x86_64
gcc-c++.x86_64
gcc-gfortran.x86_64
lz4.x86_64
cyrus-sasl-lib.x86_64
cyrus-sasl-plain.x86_64
make.x86_64
zlib.x86_64
zlib-devel.x86_64

Dependencies:
OpenSSL: 1.0.2k-fips
AWS CLI: 2.9.17
Node: v16.19.0
Python: 3.7.16

Node process versions
```
{
node: '16.19.0',
v8: '9.4.146.26-node.24',
uv: '1.43.0',
zlib: '1.2.11',
brotli: '1.0.9',
ares: '1.18.1',
modules: '93',
nghttp2: '1.47.0',
napi: '8',
llhttp: '6.0.10',
openssl: '1.1.1s+quic',
cldr: '41.0',
icu: '71.1',
tz: '2022f',
unicode: '14.0',
ngtcp2: '0.8.1',
nghttp3: '0.7.0'
}
```

My producer code:
```ts
import * as rdkafka from 'node-rdkafka';
import * as fs from 'fs';
import * as path from 'path';

const brokerString: string = 'broker-1:9094,broker-2:9094,broker-3:9094';
const topic: string = 'topic1';
const event = {
...fields
}

const producer: rdkafka.Producer = new rdkafka.Producer({
'metadata.broker.list': brokerString,
'debug': 'all',
'client.id': 'testing',
'dr_cb': true,
'security.protocol': 'ssl',
'ssl.ca.pem': fs.readFileSync(path.resolve(__dirname, '../certs/client.signed.crt'), 'utf-8'),
'ssl.certificate.pem': fs.readFileSync(path.resolve(__dirname, '../certs/client.crt'), 'utf-8'),
'ssl.key.pem': fs.readFileSync(path.resolve(__dirname, '../certs/client.key'), 'utf-8'),
'ssl.key.password': 'password'
});
console.log(`Created producer >>>> ${producer}`);

const publishEvent = () => {
console.log(`Producer started at ${new Date().toISOString()}`);
const eventString: string = JSON.stringify(event);
try {
producer.produce(topic, undefined, Buffer.from(eventString), undefined, Date.now());
} catch (e) {
console.error(e);
}
}

producer.connect();
producer.on('ready', publishEvent);
producer.on('event.error', (e) => { console.error(e); });
producer.on('event.log', (l) => { console.log(l); });
```

I've been trying to use this package in AWS, but building and trying to connect results in a segmentation fault due to the conflicting versions of openssl that are in use (node using v1.1.1 and `node-rdkafka` being built with v1.0.2). I've upgraded openssl to version 1.1.1 to hopefully avoid the segmentation fault, but then the build fails with a plethora of `node-gyp` build errors.

I've tested multiple packages for interacting with Kafka using node and this one far outpaces the others, but the build issues and segmentation faults are holding me back from being able to implement. I'd consider upgrading the package to build correctly against openssl1.1.1 would be a worthwhile improvement, since more modern systems will be using later versions.

If this has been resolved already, how do I bypass these issues and build correctly? It's a requirement in my org to use TLS when sending dat to Kafka, so I can't just ignore the openssl issues. I also don't want to downgrade my node version to where the openssl node process is compatible with `node-rdkafka` because everything else in our architecture runs on node16+.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the node-rdkafka build and TLS connection on Amazon Linux 2 with Node v16.19.0 and the listed OpenSSL versions, then inspect the node-gyp build errors and segmentation fault. Done means the package builds successfully and the producer connects over TLS without crashing.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
backend, distributed-systems, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.