[BUG] Systemd - service remains unavailable even after installation

Description

In a complex state we try to stop a service through service.dead but at this moment the service is not available. Later we install the service and try to start it through service.running. We still get an error wich indicates that the service is not available.

Setup

KVM (Proxmox) / Debian 10 / Salt 3000.9
Virtualbox (host Win10) / Debian 10 / Salt 3000.9
Virtualbox (host Win10) / Debian 10 / Salt 3005.1
Virtualbox (host Win10) / Debian 11 / Salt 3005.1

Steps to Reproduce the behavior

tomcat_service_dead:
  service.dead:
    - name: tomcat9

tomcat_install:
  pkg.installed:
    - pkgs:
      - openjdk-11-jre
      - tomcat9

tomcat_service_running:
  service.running:
    - name: tomcat9

Output:

local:
----------
          ID: tomcat_service_dead
    Function: service.dead
        Name: tomcat9
      Result: True
     Comment: The named service tomcat9 is not available
     Started: 10:19:55.458862
    Duration: 37.953 ms
     Changes:
----------
          ID: tomcat_install
    Function: pkg.installed
      Result: True
     Comment: 2 targeted packages were installed/updated.
     Started: 10:19:58.332754
    Duration: 67072.296 ms
     Changes:
              ----------
              at-spi2-core:
                  ----------
                  new:
                      2.38.0-4
                  old:
 ....
              x11-utils:
                  ----------
                  new:
                      7.7+5
                  old:
----------
          ID: tomcat_service_running
    Function: service.running
        Name: tomcat9
      Result: False
     Comment: The named service tomcat9 is not available
     Started: 10:21:05.491578
    Duration: 26.125 ms
     Changes:

Summary for local
------------
Succeeded: 2 (changed=1)
Failed:    1
------------
Total states run:     3
Total run time:  67.136 s

In the shell, we got:

 systemctl status tomcat9
● tomcat9.service - Apache Tomcat 9 Web Application Server
     Loaded: loaded (/lib/systemd/system/tomcat9.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-03-06 10:20:55 UTC; 1min 43s ago
       Docs: https://tomcat.apache.org/tomcat-9.0-doc/index.html
    Process: 20612 ExecStartPre=/usr/libexec/tomcat9/tomcat-update-policy.sh (code=exited, status=0/SUCCESS)
   Main PID: 20616 (java)
      Tasks: 29 (limit: 524)
     Memory: 72.0M
        CPU: 6.751s
     CGroup: /system.slice/tomcat9.service
             └─20616 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.>

Mar 06 10:20:57 minion113005 tomcat9[20616]: OpenSSL successfully initialized [OpenSSL 1.1.1k  25 Mar 2021]
Mar 06 10:20:59 minion113005 tomcat9[20616]: Initializing ProtocolHandler ["http-nio-8080"]
Mar 06 10:20:59 minion113005 tomcat9[20616]: Server initialization in [3181] milliseconds
Mar 06 10:20:59 minion113005 tomcat9[20616]: Starting service [Catalina]
Mar 06 10:20:59 minion113005 tomcat9[20616]: Starting Servlet engine: [Apache Tomcat/9.0.43 (Debian)]
Mar 06 10:20:59 minion113005 tomcat9[20616]: Deploying web application directory [/var/lib/tomcat9/webapps/ROOT]
Mar 06 10:21:03 minion113005 tomcat9[20616]: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were s>
Mar 06 10:21:03 minion113005 tomcat9[20616]: Deployment of web application directory [/var/lib/tomcat9/webapps/ROOT] has finished in [3,854] ms
Mar 06 10:21:03 minion113005 tomcat9[20616]: Starting ProtocolHandler ["http-nio-8080"]
Mar 06 10:21:03 minion113005 tomcat9[20616]: Server startup in [4212] milliseconds

Expected behavior

tomcat_service_running doesn't fail.

Versions Report

Salt Version:
          Salt: 3005.1

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-10-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

Additional context

It seems that this bug is related with the use of the context in the function _systemctl_status(name) in file systemd_service.py (https://github.com/saltstack/salt/blob/c20115cc17db4143818d58ccfd6b6049e688b364/salt/modules/systemd_service.py#L359)

If I replace function

def _systemctl_status(name):
    """
    Helper function which leverages __context__ to keep from running 'systemctl
    status' more than once.
    """
    contextkey = "systemd._systemctl_status.%s" % name
    if contextkey in __context__:
        return __context__[contextkey]
    __context__[contextkey] = __salt__["cmd.run_all"](
        _systemctl_cmd("status", name),
        python_shell=False,
        redirect_stderr=True,
        ignore_retcode=True,
    )
    return __context__[contextkey]

by this :

def _systemctl_status(name):
    """
    Helper function which leverages __context__ to keep from running 'systemctl
    status' more than once.
    """
    contextkey = "systemd._systemctl_status.%s" % name
    # Skip looking in context
    #if contextkey in __context__:
    #    return __context__[contextkey]
    __context__[contextkey] = __salt__["cmd.run_all"](
        _systemctl_cmd("status", name),
        python_shell=False,
        redirect_stderr=True,
        ignore_retcode=True,
    )
    return __context__[contextkey]

Everything works as expected.

Maybe we can add a use_context kwargs similarly to the use_context in the package management ? (for example https://github.com/saltstack/salt/blob/6c2b2f94adae40cc4b0f99e14f2c1add82f5ea3a/salt/modules/aptpkg.py#L1518)