Kmesh L4 Authorization sometimes doesn't work

**What happened**:

I just follow the doc: https://kmesh.net/en/docs/userguide/tcp_authorization/ to test Kmesh L4 Authorization.

It may work fine at first, but it may not work properly after restarting some pods. After that no matter how you modify the AuthorizationPolicy configuration and restart the corresponding pods, all connections will be passed.

What is even more strange is that for those connections should be denied, there indeed relevant logs in the Kmesh daemon, but the connection is actually allowed.

```bash
time="2024-07-30T03:12:32Z" level=info msg="Auth denied for connection: {srcIdentity:{trustDomain: namespace:default serviceAccount:sleep} dstNetwork: srcIp:[10 244 1 33] dstIp:[10 244 1 32] dstPort:80}" subsys=pkg/auth
```

**What you expected to happen**:

The connection should be allowed or denied based on the configuration.

**How to reproduce it (as minimally and precisely as possible)**:

Restart some pods and change the AuthorizationPolicy to match. Once it fails, it will never work again.

**Anything else we need to know?**:

**Environment**:
- Kmesh version:
- Others:

Start with the TCP Authorization guide at https://kmesh.net/en/docs/userguide/tcp_authorization/ and reproduce the failure after restarting pods and changing AuthorizationPolicy. Compare the daemon's denied-connection logs with the connections that are actually allowed, and record the Kmesh version and environment. Done means authorization decisions consistently match the policy after restarts.

Written by the indexing model from the issue text.