hashicorp / hashicorp/consul

Lots of ACL errors due to SI tokens not being found

Open
#7,441 8 comments 4 reactions 0 assignees View on GitHub
theme/consul-nomad type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

Hi, we have a cluster with consul 1.7.1 and nomad 0.10.4, and we're testing connect with ACL in that environment. After some tests (everything worked perfectly), we're seeing the following:

Client Logs:
```
Mar 12 07:26:07 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:26:07.660688Z","error":"rpc error making call: rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.16","Port":8300,"Zone":""}}
Mar 12 07:26:30 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:26:30.995703Z","error":"rpc error making call: rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.5","Port":8300,"Zone":""}}
Mar 12 07:26:38 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:26:38.383670Z","error":"rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.11","Port":8300,"Zone":""}}
Mar 12 07:26:53 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:26:53.285521Z","error":"rpc error making call: rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.5","Port":8300,"Zone":""}}
Mar 12 07:26:54 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:26:54.610362Z","error":"rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.11","Port":8300,"Zone":""}}
Mar 12 07:27:09 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:27:09.359415Z","error":"rpc error making call: rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.16","Port":8300,"Zone":""}}
Mar 12 07:27:31 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:27:31.795928Z","error":"rpc error making call: rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.5","Port":8300,"Zone":""}}
Mar 12 07:27:40 wrkr-dd379243-3b consul: {"@level":"error","@message":"RPC failed to server","@module":"agent.client","@timestamp":"2020-03-12T07:27:40.164076Z","error":"rpc error making call: ACL not found","method":"Intention.Match","server":{"IP":"10.172.0.11","Port":8300,"Zone":""}}
```

These log messages are appearing in some nodes at a rate of several per minute. I added some more log, compiled and changed a server node, and I'm seeing Intention.Match requests made with tokens that were already deleted (previous envoy nomad tasks):

![log](https://user-images.githubusercontent.com/6938602/76498156-a064d700-643c-11ea-98a3-931b312be9a9.png)

Server logs:
```
Mar 12 07:32:42 ctrl-fa10eb42-3b consul: {"@level":"warn","@message":"Match information","@module":"agent.server.intentions","@timestamp":"2020-03-12T07:32:42.113348Z","intention":"","token":"723aefc1-855c-906d-9df5-cfad94c91a63"}
Mar 12 07:32:46 ctrl-fa10eb42-3b consul: {"@level":"warn","@message":"Match information","@module":"agent.server.intentions","@timestamp":"2020-03-12T07:32:46.093700Z","intention":"","token":"fb4a896d-86af-9837-addf-1777363edfea"}
Mar 12 07:33:20 ctrl-fa10eb42-3b consul: {"@level":"warn","@message":"Match information","@module":"agent.server.intentions","@timestamp":"2020-03-12T07:33:20.664108Z","intention":"","token":"f2c68a05-0436-4ba8-b3b0-0f8c0e3ee7cf"}
Mar 12 07:33:22 ctrl-fa10eb42-3b consul: {"@level":"warn","@message":"Match information","@module":"agent.server.intentions","@timestamp":"2020-03-12T07:33:22.373436Z","intention":"","token":"6175af82-dea3-2487-b4d7-7b66ef4ffe49"}
```
After rebooting consul these log messages stop appearing.

I don't know if this is related to the cache the agents maintain, but I don't think these messages should appear constantly.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.