Lots of ACL errors due to SI tokens not being found
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
Hi, we have a cluster with consul 1.7.1 and nomad 0.10.4, and we're testing connect with ACL in that environment. After some tests (everything worked perfectly), we're seeing the following:
Client Logs:
```
Mar 12 07:26:07 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:26:07.660688Z","error":"rpc error making call: rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.16","Port":8300,"Zone":""}}
Mar 12 07:26:30 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:26:30.995703Z","error":"rpc error making call: rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.5","Port":8300,"Zone":""}}
Mar 12 07:26:38 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:26:38.383670Z","error":"rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.11","Port":8300,"Zone":""}}
Mar 12 07:26:53 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:26:53.285521Z","error":"rpc error making call: rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.5","Port":8300,"Zone":""}}
Mar 12 07:26:54 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:26:54.610362Z","error":"rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.11","Port":8300,"Zone":""}}
Mar 12 07:27:09 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:27:09.359415Z","error":"rpc error making call: rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.16","Port":8300,"Zone":""}}
Mar 12 07:27:31 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:27:31.795928Z","error":"rpc error making call: rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.5","Port":8300,"Zone":""}}
Mar 12 07:27:40 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:27:40.164076Z","error":"rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.11","Port":8300,"Zone":""}}
```
These log messages are appearing in some nodes at a rate of several per minute. I added some more log, compiled and changed a server node, and I'm seeing Intention.Match requests made with tokens that were already deleted (previous envoy nomad tasks):

Server logs:
```
Mar 12 07:32:42 ctrl-fa10eb42-3b consul: {"@level":"warn","@message":"Match information","@module":"agent.server.intentions","@timestamp":"2020-03-12T07:32:42.113348Z","intention":"","token":"723aefc1-855c-906d-9df5-cfad94c91a63"}
Mar 12 07:32:46 ctrl-fa10eb42-3b consul: {"@level":"warn","@message":"Match information","@module":"agent.server.intentions","@timestamp":"2020-03-12T07:32:46.093700Z","intention":"","token":"fb4a896d-86af-9837-addf-1777363edfea"}
Mar 12 07:33:20 ctrl-fa10eb42-3b consul: {"@level":"warn","@message":"Match information","@module":"agent.server.intentions","@timestamp":"2020-03-12T07:33:20.664108Z","intention":"","token":"f2c68a05-0436-4ba8-b3b0-0f8c0e3ee7cf"}
Mar 12 07:33:22 ctrl-fa10eb42-3b consul: {"@level":"warn","@message":"Match information","@module":"agent.server.intentions","@timestamp":"2020-03-12T07:33:22.373436Z","intention":"","token":"6175af82-dea3-2487-b4d7-7b66ef4ffe49"}
```
After rebooting consul these log messages stop appearing.
I don't know if this is related to the cache the agents maintain, but I don't think these messages should appear constantly.
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.