mild /etc/hosts pollution breaks communication between erchef and bookshelf
Nobody has claimed this yet.
- Dominant language
- Erlang
- Stars
- 303
- Forks
- 211
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 5
Description
We have a bootstrap process for pre-packaging Chef server nodes. On some RHEL7/CentOS 7[1] nodes with 12.15.7-1, we get internal server errors uploading cookbooks during this process, with what should be a reasonable configuration. The salient difference seems to be some extraneous stuff in /etc/hosts that in theory shouldn't matter...
Symptoms resemble this guy's problem: https://github.com/chef/chef/issues/2409, the tl;dr summary of which is that erchef will faceplant trying to checksum things when api_fqdn is set to something unresolvable. His logs, like mine, show stuff like this:
opscode-erchef/erchef.log:2017-06-22 18:44:26.491 [error] Checking presence of checksum: <<"2236655cd281e2a508a871c9f88db493">> for org <<"7bb5fbf89c846e2b3abc1238357afbdd">> from bucket "bookshelf" has taken longer than 5000 m
Where my issue differs is that there shouldn't be any issue resolving api_fqdn at this point, because... well, here's the chef-server.rb we're working from at this point in the process (right after a successful reconfigure/restart):
server_name='127.0.0.1'
api_fqdn server_name
nginx['server_name'] = server_name
nginx['enable_non_ssl'] = false
nginx['non_ssl_port'] = 81
nginx['ssl_port'] = 7443
nginx['ssl_ciphers'] = 'HIGH:MEDIUM:!LOW:!kEDH:!aNULL:!ADH:!eNULL:!EXP:!SSLv2:!SEED:!CAMELLIA:!PSK'
nginx['ssl_protocols'] = 'TLSv1.2'
bookshelf['external_url'] = 'https://127.0.0.1:7443'
bookshelf['vip_port'] = 7443
Now, the symptomatic machines come with an /etc/hosts that look like this:
::1 localhost6.localdomain6 localhost6
172.31.17.253 MU-PROD-2017061914-SH-CENTOS7
127.0.0.1 localhost
That second line there is a junk value inherited from the original machine used to make the image. If I remove it and do a chef-server-ctl restart, the problem goes away. During this point in the bootstrap process, our machine still thinks its name is MU-PROD-2017061914-SH-CENTOS7, so here's my theory: Erchef is picking up $HOSTNAME from somewhere and trying to use it to talk to Bookshelf, instead of honoring the value of api_fqdn. Since the IP in the hosts file is some random garbage, it times out, and wackiness ensues.
It's easy enough to work around (build cleaner images, squish /etc/hosts, whatever), but this smells like correctable brittleness.
[1] I mention the RHEL7 flavor because it does not occur on identically configured CentOS 6 nodes. I dunno what to make of that.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how erchef resolves or selects the endpoint used to communicate with bookshelf, using the reported erchef log message and the chef-server.rb settings as the entry points. Reproduce the cookbook upload failure with the shown /etc/hosts entry, then compare it with the configuration after that entry is removed. Done means the configured api_fqdn and bookshelf URL are honored despite unrelated hostname data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100