chef / chef/chef-server

mild /etc/hosts pollution breaks communication between erchef and bookshelf

Open
#1,319 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Aspect: Packaging Component: opscode-erchef Status: To be prioritized Triage: Needs Information Type: Bug
Dominant language
Erlang
Stars
303
Forks
211
Avg merge
1d 8h
Merged PRs (30d)
5

Description

We have a bootstrap process for pre-packaging Chef server nodes. On some RHEL7/CentOS 7[1] nodes with 12.15.7-1, we get internal server errors uploading cookbooks during this process, with what should be a reasonable configuration. The salient difference seems to be some extraneous stuff in /etc/hosts that in theory shouldn't matter...

Symptoms resemble this guy's problem: https://github.com/chef/chef/issues/2409, the tl;dr summary of which is that erchef will faceplant trying to checksum things when api_fqdn is set to something unresolvable. His logs, like mine, show stuff like this:

opscode-erchef/erchef.log:2017-06-22 18:44:26.491 [error] Checking presence of checksum: <<"2236655cd281e2a508a871c9f88db493">> for org <<"7bb5fbf89c846e2b3abc1238357afbdd">> from bucket "bookshelf" has taken longer than 5000 m

Where my issue differs is that there shouldn't be any issue resolving api_fqdn at this point, because... well, here's the chef-server.rb we're working from at this point in the process (right after a successful reconfigure/restart):

server_name='127.0.0.1'
api_fqdn server_name
nginx['server_name'] = server_name
nginx['enable_non_ssl'] = false
nginx['non_ssl_port'] = 81
nginx['ssl_port'] = 7443
nginx['ssl_ciphers'] = 'HIGH:MEDIUM:!LOW:!kEDH:!aNULL:!ADH:!eNULL:!EXP:!SSLv2:!SEED:!CAMELLIA:!PSK'
nginx['ssl_protocols'] = 'TLSv1.2'
bookshelf['external_url'] = 'https://127.0.0.1:7443'
bookshelf['vip_port'] = 7443

Now, the symptomatic machines come with an /etc/hosts that look like this:

::1             localhost6.localdomain6 localhost6
172.31.17.253 MU-PROD-2017061914-SH-CENTOS7
127.0.0.1 localhost

That second line there is a junk value inherited from the original machine used to make the image. If I remove it and do a chef-server-ctl restart, the problem goes away. During this point in the bootstrap process, our machine still thinks its name is MU-PROD-2017061914-SH-CENTOS7, so here's my theory: Erchef is picking up $HOSTNAME from somewhere and trying to use it to talk to Bookshelf, instead of honoring the value of api_fqdn. Since the IP in the hosts file is some random garbage, it times out, and wackiness ensues.

It's easy enough to work around (build cleaner images, squish /etc/hosts, whatever), but this smells like correctable brittleness.

[1] I mention the RHEL7 flavor because it does not occur on identically configured CentOS 6 nodes. I dunno what to make of that.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how erchef resolves or selects the endpoint used to communicate with bookshelf, using the reported erchef log message and the chef-server.rb settings as the entry points. Reproduce the cookbook upload failure with the shown /etc/hosts entry, then compare it with the configuration after that entry is removed. Done means the configured api_fqdn and bookshelf URL are honored despite unrelated hostname data.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.