hashicorp / hashicorp/nomad

after restart nomad cleint can't start again

Open
#2,715 3 comments 1 reaction 0 assignees View on GitHub
stage/needs-verification theme/client theme/client-restart type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.6.0-dev (52ffc01972a1babf0ece579e3de42a9e0c7bf8e6+CHANGES)

We try to restart nomad agent(client) on one of our servers and after that it can't start with tons of errors:

```
==> Caught signal: terminated
Loaded configuration from /etc/nomad/nomad.json
==> Starting Nomad agent...
==> Error starting agent: client setup failed: failed to restore state: 836 error(s) occurred:

* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
* failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
```

and another type of errors:

```
2017/06/16 16:44:53.846436 [WARN] fingerprint.network: Unable to parse Speed in output of '/sbin/ethtool eth0.109'
2017/06/16 16:44:57.870764 [WARN] driver.raw_exec: raw exec is enabled. Only enable if needed
2017/06/16 16:44:57.871488 [ERR] client: failed to restore state for alloc 00d68f09-f7f6-a1e7-9e5b-a559975904f5: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.871522 [ERR] client: failed to restore state for alloc 00d9ad1a-2714-912f-76e9-91f6f6fde103: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.871550 [ERR] client: failed to restore state for alloc 014e20d0-ecd7-89cc-abdb-8db9d0a98900: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.871981 [ERR] client: failed to restore state for alloc 0151a8a5-f285-50b8-cc43-65e936967a1b: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.872189 [ERR] client: failed to restore state for alloc 02098646-9174-8e72-1673-dd967da5f632: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.872235 [ERR] client: failed to restore state for alloc 0239ac96-a7a1-18da-3fe3-cdff88458d8a: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.872794 [ERR] client: failed to restore state for alloc 027fe98b-fc17-0ad2-c864-aadacc04f726: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.873101 [ERR] client: failed to restore state for alloc 02dcc355-f907-93ff-b197-b363f2d9f12d: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.873156 [ERR] client: failed to restore state for alloc 035c3af1-2737-0d81-4ca4-186d3756b8fc: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.873691 [ERR] client: failed to restore state for alloc 03ba54bb-acf8-8d8f-5bc7-7da4a2a69b76: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.873723 [ERR] client: failed to restore state for alloc 03d268e8-e345-895b-7dde-eda5f77f851f: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.873761 [ERR] client: failed to restore state for alloc 04017d07-4c9c-5721-bd19-c83220b6f1d0: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874142 [ERR] client: failed to restore state for alloc 0411fe40-5418-b206-52ef-ed4d217d57fd: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874191 [ERR] client: failed to restore state for alloc 04251de1-63c1-6d90-8f09-85196748928d: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874256 [ERR] client: failed to restore state for alloc 049f3c2e-313d-a164-d922-7724c2f1901b: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874288 [ERR] client: failed to restore state for alloc 04a18531-f021-0242-de01-13aa4d3b8b62: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874348 [ERR] client: failed to restore state for alloc 04ebf645-f090-c2d1-49c4-deec71104281: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874399 [ERR] client: failed to restore state for alloc 04f6a563-e68e-38d9-8078-2857c5c54b82: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874436 [ERR] client: failed to restore state for alloc 054ed0fd-c270-4734-664f-078bd024b4bf: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874461 [ERR] client: failed to restore state for alloc 055c3166-7475-da9a-1de6-ddb1da774fd6: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874502 [ERR] client: failed to restore state for alloc 056e4b82-78a8-08af-142c-996c18fc7bae: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874539 [ERR] client: failed to restore state for alloc 05ad35aa-eed7-20e2-77bf-95f8b1cf368c: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874583 [ERR] client: failed to restore state for alloc 05d7cab8-131d-096d-dace-9149b4b7704b: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874637 [ERR] client: failed to restore state for alloc 064ace42-6430-ac94-4170-82548ba37a68: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874662 [ERR] client: failed to restore state for alloc 06a2f777-764b-c137-3ea3-d8368e8663b5: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.874688 [ERR] client: failed to restore state for alloc 06b58602-e50c-2200-0834-0fdeafe2016a: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.876837 [ERR] client: failed to restore state for alloc 06cd8811-7fb9-8432-819e-f735d30d485e: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.876882 [ERR] client: failed to restore state for alloc 07608cd7-8868-919d-5f64-876649fdf92d: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.876927 [ERR] client: failed to restore state for alloc 0766fedf-6753-1461-17d0-a8a50c79ec1a: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
2017/06/16 16:44:57.876942 [ERR] client: failed to restore state for alloc 07f4e779-2841-9221-5654-d0f0e0779c26: failed to read allocation state: failed to read alloc runner immutable state: no data at key immutable
```

and after that all repears again and agian

It seems that this problem is the same as https://github.com/hashicorp/nomad/issues/2328

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Nomad client setup and allocation-state restoration path described by the startup error, then compare the behavior with issue #2328. Reproduce an agent restart using the reported conditions; done means the client starts again and restores allocations without repeated missing immutable-state errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.