Cannot stop multi-cpu Tornado service on FreeBSD

Start with the FreeBSD service file and the Python _main, shutdown, and exit_handler entry points shown in the report. Reproduce a multi-CPU start and stop on FreeBSD, then trace the daemon and worker process lifecycle; done means stopping the service removes every worker and releases the port for restart.

I'm unable to stop a Tornado service on FreeBSD when using multi-cpu.
When I do service appl start, I see five processes:

• one daemon
• four non-daemon threads

The four non-daemon threads are what I see when I run systemctl start appl on Ubuntu. FreeBSD has the one extra daemon thread.

When I try to stop the FreeBSD service (service app stop) the daemon process and the first (lowest pid) non-daemon process are removed. Three non-daemon processes remain running. Attempting to restart the app says the port is already in use.

I have a shutdown process which logs 'Stopped'. This does not log anything when using three CPUs though it logs properly when using one CPU. The Tornado process that spawns the three CPU worker processes is not killing them.

Again, this works just fine on Ubuntu (although the additional daemon process is missing).

Starting Tornado

My startup application code.

def make_app():
    """Create application instance."""
    args = parse_args(APPNAME, VERSION)
    config = app_config(args)
    if is_pid_running(args):
        raise AlreadyRunningError()
    create_pid(args)
    return tornado.web.Application(
        [
            (r"/payments/v2/", PaymentsHandler),
            (r"/vault/v2/", VaultHandler),
        ],
        **config,
    )


def _main():
    """Create, configure, and start the application."""
    # http://www.tornadoweb.org/en/stable/guide/running.html#running-and-deploying
    global APP, SERVER  # pylint: disable=global-statement

    app = APP = make_app()
    server = SERVER = tornado.httpserver.HTTPServer(
        app,
        xheaders=True,
    )
        server.start(3)

    ioloop = tornado.ioloop.IOLoop.current()

    signal.signal(signal.SIGTERM, exit_handler)
    signal.signal(signal.SIGINT, exit_handler)
    signal.signal(signal.SIGHUP, exit_handler)
    ioloop.start()

async def shutdown():
    """Handle graceful shutdown."""
    if APP:
        settings = APP.settings
        logger = settings["logger"]
        logger.info("Stopping.")
        SERVER.stop()
        # await tornado.gen.sleep(sleep)
        tornado.ioloop.IOLoop.current().stop()
        delete_pid(APP.settings["args"].pid_file)

def exit_handler(sig, frame):
    """Install exit handler."""
    _ = sig
    _ = frame
    tornado.ioloop.IOLoop.instance().add_callback_from_signal(shutdown)

FreeBSD Service file

name=app
RUNUSER="${name}"
RUNGROUP="${name}"

. /etc/rc.subr

rcvar=app_enable
load_rc_config $name

# Set some defaults
app_enable=${app_enable:-"NO"}
app_run_user=${app_run_user:-"$RUNUSER"}
app_run_group=${app_run_group:-"$RUNGROUP"}
app_binary="/usr/local/bin/${name}.pyz"

#pidfile=${app_pidfile:-"/var/run/${name}/${name}.pid"}
pidfile=${app_pidfile:-"/var/run/${name}.pid"}
command="/usr/sbin/daemon"

# -f redirects stdout and stderr to /dev/null
# Don't do that so we can better see start-up errors
# before logs are active.
# TODO: Need to figure out how to restart multi-cpu service.
command_args="-P ${pidfile} -u ${app_run_user} ${app_binary}"
run_rc_command "$1"

Any thoughts?

Versions

• Tornado 6.1
• FreeBSD 12.2
• Python 3.9.4