Celery integration blocks running django commands

Open
#750 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
django, python
Domain
backend

Research direction

Start by reproducing the hang with manage.py and a Django command, then trace the integration from common/celery.py into scout_apm/celery.py, focusing on the install path shown in the traceback. Done means Django management commands complete without blocking when Celery integration is enabled.

Written by the indexing model from the issue text.

Description

When I try to run any django command it gets stuck and if I interrupt it this is the traceback I get. I enabled debug logs in case it's useful

DEBUG:scout_apm.core.config:Configuration Loaded:
DEBUG:scout_apm.core.config:Defaults : app_server =
DEBUG:scout_apm.core.config:Defaults : application_root = /app
DEBUG:scout_apm.core.config:Defaults : collect_remote_ip = True
DEBUG:scout_apm.core.config:Null     : core_agent_config_file = None
DEBUG:scout_apm.core.config:Defaults : core_agent_dir = /tmp/scout_apm_core
DEBUG:scout_apm.core.config:Defaults : core_agent_download = True
DEBUG:scout_apm.core.config:Defaults : core_agent_launch = True
DEBUG:scout_apm.core.config:Null     : core_agent_log_file = None
DEBUG:scout_apm.core.config:Defaults : core_agent_log_level = info
DEBUG:scout_apm.core.config:Defaults : core_agent_permissions = 700
DEBUG:scout_apm.core.config:Defaults : core_agent_socket_path = tcp://127.0.0.1:6590
DEBUG:scout_apm.core.config:Defaults : core_agent_version = v1.4.0
DEBUG:scout_apm.core.config:Defaults : disabled_instruments = []
DEBUG:scout_apm.core.config:Defaults : download_url = https://s3-us-west-1.amazonaws.com/scout-public-downloads/apm_core_agent/release
DEBUG:scout_apm.core.config:Defaults : framework =
DEBUG:scout_apm.core.config:Defaults : framework_version =
DEBUG:scout_apm.core.config:Defaults : hostname = None
DEBUG:scout_apm.core.config:Null     : ignore = None
DEBUG:scout_apm.core.config:Null     : log_level = None
DEBUG:scout_apm.core.config:Defaults : log_payload_content = False
DEBUG:scout_apm.core.config:Python   : monitor = True
DEBUG:scout_apm.core.config:Python   : name = testtest
DEBUG:scout_apm.core.config:Python   : revision_sha = 7c7c4ecc708cf45338d150b59ac654245d546877
DEBUG:scout_apm.core.config:Python   : scm_subdirectory = /core
DEBUG:scout_apm.core.config:Defaults : shutdown_message_enabled = True
DEBUG:scout_apm.core.config:Defaults : shutdown_timeout_seconds = 2.0
DEBUG:scout_apm.instruments.elasticsearch:Instrumenting elasticsearch.
DEBUG:scout_apm.instruments.elasticsearch:Couldn't import elasticsearch.Elasticsearch - probably not installed.
DEBUG:scout_apm.instruments.elasticsearch:Couldn't import elasticsearch.Transport or elastic_transport.Transport - probably not installed.
DEBUG:scout_apm.instruments.jinja2:Instrumenting Jinja2.
DEBUG:scout_apm.instruments.pymongo:Instrumenting pymongo.
DEBUG:scout_apm.instruments.pymongo:Couldn't import pymongo.Collection - probably not installed.
DEBUG:scout_apm.instruments.redis:Instrumenting redis.
DEBUG:scout_apm.instruments.urllib3:Instrumenting urllib3.
DEBUG:scout_apm.core:APM Launching on PID: 598
DEBUG:scout_apm.core.agent.manager:Core Agent manifest json: {'version': '1.4.0', 'core_agent_version': '1.4.0', 'core_agent_binary': 'core-agent', 'core_agent_binary_sha256': '1a24b1ab8a0ca7ffc831ca83614e0d4c2b4b801f4a4eec0c3cfa0a363caf6ff8'}
^CDEBUG:scout_apm.core.agent.socket:CoreAgentSocketThread attempt 1, connecting to tcp://127.0.0.1:6590, PID: 598, Thread: <CoreAgentSocketThread(Thread-1, stopped daemon 140094993512000)>
DEBUG:scout_apm.core.agent.socket:CoreAgentSocketThread connected
INFO:scout_apm.core.agent.commands:Registering with app=testtest key_prefix=XQb key_format_validated=True host=None
Traceback (most recent call last):
  File "/app/manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 395, in execute
    django.setup()
  File "/usr/local/lib/python3.10/dist-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/local/lib/python3.10/dist-packages/django/utils/log.py", line 75, in configure_logging
    logging_config_func(logging_settings)
  File "/usr/lib/python3.10/logging/config.py", line 810, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/lib/python3.10/logging/config.py", line 544, in configure
    formatters[name] = self.configure_formatter(
  File "/usr/lib/python3.10/logging/config.py", line 676, in configure_formatter
    c = _resolve(cname)
  File "/usr/lib/python3.10/logging/config.py", line 90, in _resolve
    found = __import__(used)
  File "/app/common/__init__.py", line 1, in <module>
    from common import tasks  # noqa
  File "/app/common/tasks/__init__.py", line 1, in <module>
    from .maintenance import *  # noqa
  File "/app/common/tasks/maintenance.py", line 6, in <module>
    from common.celery import app
  File "/app/common/celery.py", line 26, in <module>
    scout_celery.install(app)
  File "/usr/local/lib/python3.10/dist-packages/scout_apm/celery.py", line 133, in install
    installed = scout_apm.core.install()
  File "/usr/local/lib/python3.10/dist-packages/scout_apm/compat.py", line 141, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/scout_apm/core/__init__.py", line 50, in install
    CoreAgentSocketThread.ensure_stopped()
  File "/usr/local/lib/python3.10/dist-packages/scout_apm/core/threading.py", line 44, in ensure_stopped
    cls._instance.join()DEBUG:scout_apm.core.tracked_request:Starting request: req-f0c17d8b-7f2b-47c0-b74c-3a81ab67f01f

  File "/usr/lib/python3.10/threading.py", line 1096, in join
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): sentry.*****.****:443
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
KeyboardInterrupt
DEBUG:urllib3.connectionpool:https://sentry.*****.****:443 "POST /api/2/store/ HTTP/1.1" 200 41
DEBUG:scout_apm.core.tracked_request:Stopping request: req-f0c17d8b-7f2b-47c0-b74c-3a81ab67f01f
DEBUG:scout_apm.core.tracked_request:Request req-f0c17d8b-7f2b-47c0-b74c-3a81ab67f01f start_time=2023-06-01 17:17:25.078999 end_time=2023-06-01 17:17:25.162968 duration=0.083969 active_spans=0 complete_spans=1 tags=0 hit_max=False is_real_request=False sent=False
Dominant language
Python
Stars
77
Forks
19
Avg merge
1d 9h
Merged PRs (30d)
5

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from scoutapp/scout_apm_python

All issues in scoutapp/scout_apm_python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.