pinpoint-apm / pinpoint-apm/pinpoint-cpp-agent
fork-safe agent lifecycle
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Summary
The agent is not safe to use across fork(). The most common Python (and many
C++/PHP) deployment models — gunicorn, uWSGI, multiprocessing(fork) —
initialize the agent in a master process and then fork worker processes. Because
the agent's gRPC worker threads and gRPC runtime do not survive fork(), forked
workers inherit a broken agent. Downstream bindings (e.g.
pinpoint-python-agent) currently can only work around this with fragile hacks.
This issue proposes first-class fork support in the C++ agent.
Background: what breaks after fork
-
Worker threads are dead in the child.
AgentImpl's constructor starts
init_thread_(which spawns the ping/meta/span/stat/url-stat/command workers
viainit_grpc_workers()). Threads are not duplicated acrossfork(), so a
child inherits anAgentImplwhose background threads no longer exist. Spans
enqueued in the child are silently dropped. -
Destroying the inherited agent crashes.
~AgentImpl()→do_shutdown()
→close_grpc_workers()→wait_grpc_workers()joins the worker
threads. In the child thosestd::threadhandles are joinable but reference
threads that don't exist →pthread_joinfails /std::system_errorescapes
the joiner thread →std::terminate()→ abort. This also fires via the
atexit/Shutdown()path at worker exit. -
CreateAgent()cannot rebuild.create_agent_helper()returns the
existing global agent (withreloadConfig()) whenever
global_agent() != nullptr. In a forked child the global is non-null
(inherited), soCreateAgent()just hands back the same dead agent —
there is no way to obtain a fresh, working one. -
gRPC runtime is initialized at create time.
GrpcClient::GrpcClient()
(src/grpc.cpp:186-190) callsbuild_channel()→
grpc::CreateCustomChannel(), and theGrpc*clients are constructed inside
make_agent(). Sogrpc_init()and gRPC's own background threads are up as
soon asCreateAgent()runs. Forking a process with an active gRPC stack is
deadlock-prone unless gRPC's fork handlers are enabled. -
agent_idis shared across children.generate_agent_id()
(src/config.cpp,hostname[:18]-random5) runs once at config time. If a
master generates the id and forks, all workers would register under the same
id.
Why this can't be fixed in the binding/Python layer alone
create_agent()never rebuilds while the global is set, and
Shutdown()/destruction joins dead threads → crash. The only internal escape
hatch,reset_global_agent(), is not declared in the public header
(include/pinpoint/tracer.h), and even calling it risks destroying the agent
(→ join → crash) unless the object is deliberately leaked.- As a stopgap, a downstream binding currently must: forward-declare the
internalreset_global_agent()symbol, permanently leak the inherited
AgentImpl(a never-freedshared_ptr) so its destructor never joins, then
create_agent()again, and separately setGRPC_ENABLE_FORK_SUPPORT=1from
Python. This works but depends on an unexported-by-contract symbol and manual
leaking — it belongs in the agent.
Proposed functionality
1. (Recommended) Split construction from startup — "cold create + Start()"
Make CreateAgent() cold: parse config and allocate the object only — no
threads, no gRPC channel / grpc_init, no config-file watcher. Add an
explicit startup entry point that brings the agent online in the current
process:
// Cold: config + object allocation only. Safe to call in a master that
// will fork; the returned agent holds no threads and has not touched gRPC.
AgentPtr CreateAgent(int32_t app_type, /* ...server metadata... */);
// Bring the agent online in THIS process: build gRPC channels (grpc_init),
// spawn worker threads, start the config-file watcher, and (re)generate a
// process-unique agent id. Idempotent and non-blocking (async connect).
void Agent::Start();
This enables the industry-standard "initialize per worker after fork" model
(cf. OpenTelemetry's post_fork guidance): the master calls CreateAgent(),
and each child calls Start() from its fork hook. The parent never holds live
threads or an open gRPC runtime at the fork point, which removes failure modes
1–4 at the root and makes the inherited object directly reusable (no reset/leak
needed).
Note: deferring only the pinpoint worker threads is insufficient —
grpc_initmust also move intoStart()(see failure mode 4), otherwise the
gRPC runtime is still up at fork time.
2. PID-guarded teardown (safety net)
Store owner_pid_ = getpid() in AgentImpl and, in do_shutdown()/
~AgentImpl(), detach instead of join when getpid() != owner_pid_. This
guarantees no crash even if an agent that was already started in the master gets
destroyed in a child (preload/master-tracing cases).
3. Agent owns gRPC fork support
Enable gRPC's fork handlers from within the agent before the first channel is
created (equivalent to GRPC_ENABLE_FORK_SUPPORT=1, which grpcio implements via
pthread_atfork). Since the agent owns gRPC initialization, this is the correct
layer — bindings shouldn't have to set a global env var.
4. Process-unique agent_id on start
Generate/refresh the agent_id at Start() time so each forked worker gets a
unique identity (random when unset; a pid suffix or regeneration when an
explicit id is pinned).
Acceptance criteria
- A master process can create an agent,
fork(), and have each child obtain
a fully working agent (live workers, spans delivered) without crashing or
deadlocking. - Destroying/shutting down an inherited agent in a child never joins dead
threads (no abort). - Forked workers register with distinct
agent_ids. - No global env var or unexported symbol is required from the binding.
- Behavior on non-fork platforms is unchanged.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with AgentImpl, create_agent_helper(), and make_agent(), then inspect src/grpc.cpp and src/config.cpp for gRPC initialization and agent ID generation. Review the public API in include/pinpoint/tracer.h and existing shutdown paths before designing the lifecycle changes. Done means forked children can start working agents, shut down inherited agents safely, receive distinct agent IDs, and require no binding-side environment variable or internal symbol.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, grpc
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100