elastic / elastic/crawler

Intermittent `bad_record_mac` error when running on MacOS

Open
#37 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Ruby
Stars
224
Forks
48
Avg merge
23h 16m
Merged PRs (30d)
18

Description

## Bug Description

This error comes up intermittently when running on MacOS (M1 silicon, `14.4.1`). The error occurs when attempting to bulk index docs into Elasticsearch.
This doesn't occur on Linux or when running from Docker. It only seems to occur when indexing into an ES cloud instance. When indexing into a local or dockerized Elasticsearch the error doesn't occur.

### To Reproduce
Steps to reproduce the behavior:
1. Set up Crawler on an M1 Mac
2. Configure an ES cloud instance to bulk index results into
3. Crawl a few times
4. See error

## Expected behavior

SSL/TLS should be trusted automatically and this error should never occur.

## Environment

- OS: M1 silicon, `14.4.1`
- Ruby: jruby-9.4.7.0
- Java: OpenJDK 21

## Additional context

- This can be worked around by configuring a `ca_fingerprint`, or adding a cert for ES to the Mac's OpenSSL cert dir and rehashing
- It _might_ be caused by the lone bulk indexer being called by multi-threaded crawl result handler ([see this ancient similar issue](https://download.oracle.com/javaee-archive/grizzly.java.net/users/2007/06/0128.html))
- It _might_ be caused by outdated certs on MacOS, but unlikely
- For some reason the error seems to occur less frequently if the max bulk load is reduced from `100` to a lower number like `10`, but it still sometimes occurs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.