Azure / Azure/iot-hub-device-update
SIGSEGV on every failed startup: uninit_api_svc joins a thread that was never created
- Dominant language
- C++
- Stars
- 61
- Forks
- 54
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 1
Description
### Summary
`AducIotAgent` crashes with `SIGSEGV` on **every failed startup**: whenever
`HealthCheck()` fails, the agent dies in the shutdown path instead of exiting with
a non-zero code. Seen on aarch64, 1.4.0; the same code is on `develop`.
Log right before the crash:
```
[E] Invalid connection info. [HealthCheck:630]
[E] Agent health check failed. [main:1161]
→ SIGSEGV, si_code=1 (SEGV_MAPERR), fault address 0xd0
```
Because the unit restarts the agent after a failure, this repeats until the
systemd start limit stops the service — on our devices 10 cores in 60 s while the
identity service could not hand out connection info.
### Root cause
`uninit_api_svc()` (`src/agent/api/src/apisvc.c`) joins `g_api_svc_thread`
unconditionally:
```c
atomic_store(&g_api_svc_thread_running, false);
int res = pthread_join(g_api_svc_thread, (void**)&threadRet);
```
`g_api_svc_thread` is `pthread_t g_api_svc_thread = { 0 };` and is only ever set by
`init_api_svc()`. On the health-check failure path `init_api_svc()` never runs:
`main()` does `goto done` before `StartupAgent()`, and `done:` calls
`ShutdownAgent()`, which calls `uninit_api_svc()`. So `pthread_join()` gets a NULL
thread descriptor.
glibc's join implementation dereferences the descriptor to validate it. On aarch64
that faults at `NULL + 0xd0`; x86-64 glibc happens to check the handle for NULL and
returns `ESRCH`, which is why the crash only shows on arm.
Evidence from a device core (two cores, identical):
- faulting instruction `ldr w0, [x0, #208]` with `x0 = 0`, inside the join
implementation in libc
- return address points right after `bl pthread_join@plt` in `uninit_api_svc()`
The analogous timer thread teardown already guards against this
(`_stop_thread()` in `src/utils/timer_utils/src/timer.c` only joins when
`threadCreated`); the API service thread does not.
### Reproduces when
Any shutdown that was not preceded by a successful `StartupAgent()`, i.e. every
`HealthCheck()` failure (identity service unavailable, unprovisioned device, bad
connection info). Deterministic on arm.
### Fix
Take the running flag with `atomic_exchange()` in `uninit_api_svc()` and skip the
teardown when it was already false; this also makes a repeat uninit a no-op. PR
attached.
Contributor guide
Research direction
Start in src/agent/api/src/apisvc.c and trace main()'s done: path through ShutdownAgent() into uninit_api_svc(); compare its thread teardown with _stop_thread() in src/utils/timer_utils/src/timer.c. Reproduce a HealthCheck() failure and verify the agent exits nonzero without SIGSEGV, including safe repeated teardown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- embedded-iot, operating-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100