newrelic / newrelic/newrelic-python-agent

Support fork-per-job worker frameworks (e.g. RQ) in the Python agent

Open
#1,526 1 comment 0 reactions 1 assignee View on GitHub

@TimPansino is already working on this.

Since Oct 8, 2025.

feature-request pm
Dominant language
Python
Stars
210
Forks
137
Avg merge
4d 19h
Merged PRs (30d)
12

Description

Summary

The New Relic Python agent currently assumes long-lived processes (web servers, Celery prefork).
In fork-per-job frameworks like RQ, each job runs in its own short-lived child process.
This leads to data loss and high overhead when trying to instrument these jobs with the existing agent.

Desired Behavior

  • Allow parent process (worker launcher) to call register_application() once and keep a harvester alive.
  • Child processes should be able to accept distributed trace headers, run the job, and emit lightweight job events to the parent.
  • Parent consumer should handle the actual New Relic API calls (accept_distributed_trace_headers, record_custom_event, record_exception).
  • This reduces per-job handshake cost and prevents metric/event loss when children exit with os._exit().

Possible Solution

  • Provide an official recipe for fork-per-job frameworks:
    • Parent process: start a consumer thread (UDS datagram, stdout, or Redis Streams) and register once.
    • Child process: fire-and-forget emit of {type, payload, trace_headers}.
  • Or, add an optional feature flag (e.g. fork_per_job.enabled=true) with a built-in transport adapter.
  • Maintain backward compatibility: default behavior unchanged.

Additional Context

  • Today’s workaround is calling register_application()/shutdown_agent() per job, which is both slow and lossy.
  • RQ is widely used for its simplicity and isolation guarantees, but its fork-per-job model makes New Relic integration hard.
  • This feature would let RQ users adopt New Relic APM more easily, while keeping job isolation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.