Blizzard / Blizzard/node-rdkafka
Segmentation fault and build errors using node16 and OpenSSL
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
OS:
Name: Amazon Linux
Version: 2
ID Like: centos rhel fedora
Architecture: x86_64
Installed packages (yum):
openssl.x86_64
openssl-devel.x86_64
openssl-libs.x86_64
openssl11.x86_64
openssl11-libs.x86_64
openssl11-pkcs11.x86_64
gcc.x86_64
gcc-c++.x86_64
gcc-gfortran.x86_64
lz4.x86_64
cyrus-sasl-lib.x86_64
cyrus-sasl-plain.x86_64
make.x86_64
zlib.x86_64
zlib-devel.x86_64
Dependencies:
OpenSSL: 1.0.2k-fips
AWS CLI: 2.9.17
Node: v16.19.0
Python: 3.7.16
Node process versions
```
{
node: '16.19.0',
v8: '9.4.146.26-node.24',
uv: '1.43.0',
zlib: '1.2.11',
brotli: '1.0.9',
ares: '1.18.1',
modules: '93',
nghttp2: '1.47.0',
napi: '8',
llhttp: '6.0.10',
openssl: '1.1.1s+quic',
cldr: '41.0',
icu: '71.1',
tz: '2022f',
unicode: '14.0',
ngtcp2: '0.8.1',
nghttp3: '0.7.0'
}
```
My producer code:
```ts
import * as rdkafka from 'node-rdkafka';
import * as fs from 'fs';
import * as path from 'path';
const brokerString: string = 'broker-1:9094,broker-2:9094,broker-3:9094';
const topic: string = 'topic1';
const event = {
...fields
}
const producer: rdkafka.Producer = new rdkafka.Producer({
'metadata.broker.list': brokerString,
'debug': 'all',
'client.id': 'testing',
'dr_cb': true,
'security.protocol': 'ssl',
'ssl.ca.pem': fs.readFileSync(path.resolve(__dirname, '../certs/client.signed.crt'), 'utf-8'),
'ssl.certificate.pem': fs.readFileSync(path.resolve(__dirname, '../certs/client.crt'), 'utf-8'),
'ssl.key.pem': fs.readFileSync(path.resolve(__dirname, '../certs/client.key'), 'utf-8'),
'ssl.key.password': 'password'
});
console.log(`Created producer >>>> ${producer}`);
const publishEvent = () => {
console.log(`Producer started at ${new Date().toISOString()}`);
const eventString: string = JSON.stringify(event);
try {
producer.produce(topic, undefined, Buffer.from(eventString), undefined, Date.now());
} catch (e) {
console.error(e);
}
}
producer.connect();
producer.on('ready', publishEvent);
producer.on('event.error', (e) => { console.error(e); });
producer.on('event.log', (l) => { console.log(l); });
```
I've been trying to use this package in AWS, but building and trying to connect results in a segmentation fault due to the conflicting versions of openssl that are in use (node using v1.1.1 and `node-rdkafka` being built with v1.0.2). I've upgraded openssl to version 1.1.1 to hopefully avoid the segmentation fault, but then the build fails with a plethora of `node-gyp` build errors.
I've tested multiple packages for interacting with Kafka using node and this one far outpaces the others, but the build issues and segmentation faults are holding me back from being able to implement. I'd consider upgrading the package to build correctly against openssl1.1.1 would be a worthwhile improvement, since more modern systems will be using later versions.
If this has been resolved already, how do I bypass these issues and build correctly? It's a requirement in my org to use TLS when sending dat to Kafka, so I can't just ignore the openssl issues. I also don't want to downgrade my node version to where the openssl node process is compatible with `node-rdkafka` because everything else in our architecture runs on node16+.
Contributor guide
Research direction
Start by reproducing the node-rdkafka build and TLS connection on Amazon Linux 2 with Node v16.19.0 and the listed OpenSSL versions, then inspect the node-gyp build errors and segmentation fault. Done means the package builds successfully and the producer connects over TLS without crashing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- backend, distributed-systems, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100