cloudflare / cloudflare/workerd

workerd: macOS symbolication does not work

Open
#1,335 2 comments 0 reactions 0 assignees View on GitHub
bug macos tooling
Dominant language
C++
Stars
8.7k
Forks
739
Avg merge
2d 20h
Merged PRs (30d)
174

Description

Symbolication with `llvm-symbolizer` was enabled for the workerd github runners in `test.yml` in #1247.

Shortly after, it was suspended for macOS in commit ad20f5e039d34 in PR #1283 when additional build configurations were added because it leads to test timeouts. The symbol search process appears to be slow / misconfigured and the CPU and storage capabilities of the free-tier github runners are limited.

We've had issues with macOS, symbols for LLVM, and bazel build. [workerd explicitly sets a source-map for debugging](https://github.com/cloudflare/workerd/blob/7ec997844a83aec4980dbd1da404f8185b8e86d8/.vscode/launch.json#L61-L65). The problem in this ticket is likely related.

I've spent a day or two looking into this, but have not cracked it. This ticket is just to track that we have a problem here. The next step is to collect fs activity from `llvm-symbolizer`, macOS is very different from Linux / Windows; procmon on windows is king :-). For macOS, it looks like it needs System Integrity Protection turning off and then finding a dtrace based tool for the fs activity ([opensnoop](https://ss64.com/osx/opensnoop.html) or some-such).

It would also be good to fix this as a reasonable proportion of workerd users have macOS machines and they have no chance of reporting stack traces even if they want to.

A couple of related observations:

1) macOS has it's own `backtrace_symbols` function. This might be more expedient to use since it should give meaningful stacks without needing to run an additional tool.

2) `llvm-symbolizer` has a `--filter-markup` option that consumes [symbolizer markup](https://llvm.org/docs/SymbolizerMarkupFormat.html). This probably post-dates the code used in workerd (capnproto) and might make things marginally cleaner.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.