Automated deregistration for dead nodes from catalog

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
* If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

#### Is your feature request related to a problem? Please describe.

When saving a snapshot (`consul snapshot save /tmp/abc.snap`) and restoring the snapshot into a different Kubernetes cluster (`consul snapshot restore /tmp/abc.snap`), consul has a catalog entry for dead nodes, despite active members being updated appropriately.

For example, I have some nodes in a 10.62.xx subnet:

```json
{"@level":"warn","@message":"EnsureRegistration failed","@module":"agent.fsm","@timestamp":"2022-10-01T23:02:02.273498Z","error":"failed inserting node: Error while renaming Node ID: \"13d052e0-8176-8bb8-482e-24271958b68f\": Node name consul-consul-server-2 is reserved by node a710f143-23d5-678c-a487-b9db6ea7f98e with name consul-consul-server-2 (10.62.1.53)"}
```

The catalog shows these nodes:

```shell-session
$ consul catalog nodes
Node ID Address DC
consul-consul-server-0 386fa727 10.62.1.51 dc1
consul-consul-server-1 71e732a7 10.62.1.52 dc1
consul-consul-server-2 a710f143 10.62.1.53 dc1
```

However, the member list does not (because we moved the consul installation from a Kubernetes cluster running in the 10.62 subnet to one in the 10.12 subnet):

```shell-session
$ consul members
Node Address Status Type Build Protocol DC Partition Segment
consul-consul-server-0 10.12.48.48:8301 alive server 1.13.2 2 dc1 default
consul-consul-server-1 10.12.48.157:8301 alive server 1.13.2 2 dc1 default
consul-consul-server-2 10.12.49.37:8301 alive server 1.13.2 2 dc1 default
```

The solution to the above error messages is to manually deregister nodes from the catalog, which also only appears possible using the REST API and not through the `consul` command:

```shell-session
$ curl --request PUT --data '{"Node":"consul-consul-server-0"}' -v http://localhost:8500/v1/catalog/deregister
$ curl --request PUT --data '{"Node":"consul-consul-server-1"}' -v http://localhost:8500/v1/catalog/deregister
$ curl --request PUT --data '{"Node":"consul-consul-server-2"}' -v http://localhost:8500/v1/catalog/deregister
```

After the deregistration is complete, the new nodes appear in the catalog and the log messages stop:

```shell-session
$ consul catalog nodes
Node ID Address DC
consul-consul-server-0 db1e1511 10.12.33.6 dc1
consul-consul-server-1 6aa338cf 10.12.33.131 dc1
consul-consul-server-2 13d052e0 10.12.32.6 dc1
```

#### Feature Description

Essentially, the problem is that the snapshots seem to contain the node list, and restoring the snapshot in a different cluster results in some error messages appearing.

There may be some different approaches to solve this, such as:

* Automatically deleting the catalog entry, either on conflict or after a timeout
* Avoiding the use of the hostname as a persistent node identifier (or combining it with the Kubernetes node name or similar)
* Documentation describing this step
* A `consul` CLI command to deregister nodes from the catalog (this makes things a little more convenient than running `curl` commands, but is really just making the workaround easier

#### Use Case(s)

Anyone moving a Consul installation (this one backs a Vault installation) or using the snapshot save/restore capability for backups would be affected by this problem.

#### Contributions

No