Intermittent `bad_record_mac` error when running on MacOS
- Dominant language
- Ruby
- Stars
- 224
- Forks
- 48
- Avg merge
- 23h 16m
- Merged PRs (30d)
- 18
Description
## Bug Description
This error comes up intermittently when running on MacOS (M1 silicon, `14.4.1`). The error occurs when attempting to bulk index docs into Elasticsearch.
This doesn't occur on Linux or when running from Docker. It only seems to occur when indexing into an ES cloud instance. When indexing into a local or dockerized Elasticsearch the error doesn't occur.
### To Reproduce
Steps to reproduce the behavior:
1. Set up Crawler on an M1 Mac
2. Configure an ES cloud instance to bulk index results into
3. Crawl a few times
4. See error
## Expected behavior
SSL/TLS should be trusted automatically and this error should never occur.
## Environment
- OS: M1 silicon, `14.4.1`
- Ruby: jruby-9.4.7.0
- Java: OpenJDK 21
## Additional context
- This can be worked around by configuring a `ca_fingerprint`, or adding a cert for ES to the Mac's OpenSSL cert dir and rehashing
- It _might_ be caused by the lone bulk indexer being called by multi-threaded crawl result handler ([see this ancient similar issue](https://download.oracle.com/javaee-archive/grizzly.java.net/users/2007/06/0128.html))
- It _might_ be caused by outdated certs on MacOS, but unlikely
- For some reason the error seems to occur less frequently if the max bulk load is reduced from `100` to a lower number like `10`, but it still sometimes occurs.
Contributor guide
Assessment
This issue has not been assessed yet.