Polling Sensor token has expired.

I created a polling sensor that uses a datastore key as a queue.
The poll interval is set to 10 seconds
It makes a call to the datastore service to check a key for values.
As new values are added this sensor will remove them as they age.

I setup an alert to notify myself when the sensor fails and I received this today:

2020-04-29 15:42:39,191 139660794638416 AUDIT access [-] Access granted to "sensor_service" with the token set to expire at "2020-04-30T15:42:39.189539Z". (username='sensor_service',token_expiration='202
2020-04-29 15:42:39,224 139660794638416 INFO intel_alien_sensor [-] last pull time: 2020-04-29T03:42:38.460618+00:00
2020-04-29 15:42:39,294 139660794638416 INFO intel_alien_sensor [-] count:3347
2020-04-29 15:42:40,094 139660794638416 INFO intel_alien_sensor [-] lmintel.intelAlienSensor no events found
2020-04-29 15:42:43,081 140467950239824 INFO rate_limit_queue [-] checking queues
2020-04-29 15:42:43,081 140467950239824 INFO keyvalue [-] checking rate limit queues
2020-04-29 15:42:43,090 140467950239824 WARNING sensor_wrapper [-] Sensor "RateLimitQueue" run method raised an exception: 401 Client Error: Unauthorized
MESSAGE: Unauthorized - Token has expired. for url: http://127.0.0.1:9101/v1/keys/?prefix=ratelimit2731795c-e948-11e9-81b4-2a2ae2dbcce4%3Atodo.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/container/sensor_wrapper.py", line 229, in run
    self._sensor_instance.run()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/sensor/base.py", line 119, in run
    self.poll()
  File "/opt/stackstorm/packs/lmutil27/sensors/rate_limit_queue.py", line 41, in poll
    self._client.check_todo()
  File "/opt/stackstorm/virtualenvs/lmutil27/lib/python2.7/site-packages/stackrlclient/keyvalue.py", line 314, in check_todo
    for i in self.all_todo():
  File "/opt/stackstorm/virtualenvs/lmutil27/lib/python2.7/site-packages/stackrlclient/keyvalue.py", line 271, in all_todo
    return  self._all_keyvals(RLClient.TODO)
  File "/opt/stackstorm/virtualenvs/lmutil27/lib/python2.7/site-packages/stackrlclient/keyvalue.py", line 264, in _all_keyvals
    keyvals = self.client.keys.query_with_count(prefix=prefix_str)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2client/models/core.py", line 41, in decorate
    return func(*args, **kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2client/models/core.py", line 285, in query_with_count
    instances, response = self._query_details(**kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2client/models/core.py", line 273, in _query_details
    self.handle_error(response)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2client/models/core.py", line 167, in handle_error
    response.raise_for_status()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 401 Client Error: Unauthorized
MESSAGE: Unauthorized - Token has expired. for url: http://127.0.0.1:9101/v1/keys/?prefix=ratelimit2731795c-e948-11e9-81b4-2a2ae2dbcce4%3Atodo
2020-04-29 15:42:43,091 140467950239824 ERROR (unknown file) [-] Traceback (most recent call last):

2020-04-29 15:42:43,092 140467950239824 ERROR (unknown file) [-]   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/container/sensor_wrapper.py", line 361, in <module>

2020-04-29 15:42:43,092 140467950239824 ERROR (unknown file) [-]
2020-04-29 15:42:43,092 140467950239824 ERROR (unknown file) [-] obj.run()

2020-04-29 15:42:43,092 140467950239824 ERROR (unknown file) [-]   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/container/sensor_wrapper.py", line 235, in run

2020-04-29 15:42:43,092 140467950239824 ERROR (unknown file) [-]
2020-04-29 15:42:43,092 140467950239824 ERROR (unknown file) [-] raise Exception(msg)

2020-04-29 15:42:43,093 140467950239824 ERROR (unknown file) [-] Exception
2020-04-29 15:42:43,093 140467950239824 ERROR (unknown file) [-] :
2020-04-29 15:42:43,093 140467950239824 ERROR (unknown file) [-] Sensor "RateLimitQueue" run method raised an exception: 401 Client Error: Unauthorized
MESSAGE: Unauthorized - Token has expired. for url: http://127.0.0.1:9101/v1/keys/?prefix=ratelimit2731795c-e948-11e9-81b4-2a2ae2dbcce4%3Atodo.
2020-04-29 15:42:43,093 140467950239824 ERROR (unknown file) [-]

2020-04-29 15:42:43,093 140467950239824 INFO sensor_wrapper [-] Stopping trigger watcher
2020-04-29 15:42:43,094 140467950239824 INFO sensor_wrapper [-] Invoking cleanup on sensor
2020-04-29 15:42:43,244 140255981185616 INFO process_container [-] Process for sensor lmutil27.RateLimitQueue has exited with code 1
2020-04-29 15:42:45,793 140255980311728 WARNING api [-] "auth.api_url" configuration option is not configured
2020-04-29 15:42:46,632 139688247312464 INFO sensor_wrapper [-] Found config for sensor "RateLimitQueue"
2020-04-29 15:42:46,632 139688247312464 INFO sensor_wrapper [-] Watcher started
2020-04-29 15:42:46,632 139688247312464 INFO sensor_wrapper [-] Running sensor initialization code
2020-04-29 15:42:46,633 139688247312464 INFO sensor_wrapper [-] Running sensor in active mode (poll interval=10s)
2020-04-29 15:42:46,633 139688247312464 INFO rate_limit_queue [-] checking queues
2020-04-29 15:42:46,633 139688247312464 INFO keyvalue [-] checking rate limit queues
2020-04-29 15:42:46,646 139688193976112 INFO mixins [-] Connected to amqp://guest:**@127.0.0.1:5672//

Start with packs/lmutil27/sensors/rate_limit_queue.py and the stackrlclient keyvalue.py calls shown in the traceback, then trace authentication errors through st2client/models/core.py. Reproduce the 10-second polling scenario with an expired token and determine the expected sensor behavior. Done means the RateLimitQueue sensor no longer exits when its datastore request receives a token-expired response.