Chef Server bootstrapping hurts my head

The problem

The Chef Server's bootstrap process is the source of many chef-server
bug reports. This is partially the result of the complicated process
used to determine when various bootstrap operations should be run.

Currently, the omnibus cookbooks use the following flags and helper
functions to make decisions about whether or not to bootstrap:

• OmnibusHelper.has_been_bootstrapped?
• node['private_chef']['bootstrap']['enable']
• BootstrapPreflightValidator.new(node).bypass_bootstrap?
• is_data_master? (defined in enterprise-chef-common)
• backend_secondary? (defined in enterprise chef common)

In addition to these functions, there is also a long tail of one-off
checks for specific bits of state that may or may not exist.

I at least, find it very difficult to reason about the possible
outcomes of a reconfigure given an input state. Even with just the
flags above, we have 32 different paths to consider. The ad-hoc checks
I mentioned above and the fact that the output of the functions above
can change through the course of a reconfigure exacerbated the
problem.

Why the complexity

Ultimately, the number of flags and checks we have is growing over
time because

(1) We or our users start using their chef-server in a new
way that changes the assumptions of what the state will be at install
time,

1. We have to support a number of different topologies all of which
   have different requirements about when and where bootstrapping should
   happen.
2. Developers adding features are unclear on what flags guard against
   what and often add their own check to avoid this confusion.

A possible path forward

1. I believe that with no behavior changes, the /current/ list of
   functions and checks we are using can be reduced. I've started taking
   notes on where these functions are used and have included them here.
   Many seem redundant can can likely be collapsed into a single check.

2. The "bootstrap" process is trying to achieve a number of goals:

   1. Setup basic users and credentials for data storage services
      (when we are managing those services)
   2. Setup service-specific users for data storage services (when we
      are managing them and when they are external)
   3. Apply any necessary schemas to the data storage backends
   4. Create basic Chef data structure such as the pivotal user

   Some of these can be done idempotently such that no guards are
   needed. They can be run from any node at any time. Other bits are
   harder to be done idempotently and it is important they they are
   only done on a single node. As a first step, it might be useful to
   clearly delineate those two type of operations and develop a single
   way to ensure that we only run those operations on the correct node,
   once.

Notes

What follows are some raw notes I took on the methods above when trying to debug a recent problem.

OmnibusHelper.has_been_bootstrapped?

The intent of this is to provide a way of determining if we have
already run the bootstrap process on the node. It determines this by
checking for a sentinel file. The sentinel file is written out at the
end of the bootstrap recipe. Thus, this check does reliably tell
you: "has the bootstrap recipe run to completion on this node".

# This file is touched once initial bootstrapping of the system is
# done.
  def self.bootstrap_sentinel_file
    "/var/opt/opscode/bootstrapped"
  end

  # Use the presence of a sentinel file as an indicator for whether
  # the server has already had initial bootstrapping performed.
  #
  # @todo: Is there a more robust way to determine this, i.e., based
  #   on some functional aspect of the system?
  def self.has_been_bootstrapped?
    File.exists?(bootstrap_sentinel_file)
  end

Actual uses in code

• In postgresql_validator we use it in a test to determine if the
  postgresql external flag has been changed since a previous run.

libraries/preflight_postgres_validator.rb
76:    if OmnibusHelper.has_been_bootstrapped? && backend?  &&
previous_run

• In preflight_solr_validator we use it in a test to determine if the
  solr external flag has been changed since a previous run:

libraries/preflight_solr_validator.rb
32:    if OmnibusHelper.has_been_bootstrapped? && backend?  && previous_run

• In recipes/bootstrap we use it to
  • (in the negation) guard against installing add-ons
  • guard against starting postgresql and oc_bifrost
  • guard against bootstrapping the chef-server-data (i.e. creating
    basic top level objects in the provisioned database)

recipes/bootstrap.rb
20:if (!OmnibusHelper.has_been_bootstrapped? &&
40:    not_if { OmnibusHelper.has_been_bootstrapped? }
48:  not_if { OmnibusHelper.has_been_bootstrapped? }

• In recipes/default we use it in a check to decide whether to disable
  the bootstrap recipe. Because of this, I think that all uses in the
  bootstrap recipe itself are redundant.

recipes/default.rb
50:if OmnibusHelper.has_been_bootstrapped? or

• In recipe/partybus.rb we use it to decide whether to try to run the
  upgrade or just set the initial migration level.

recipes/partybus.rb
75:  if OmnibusHelper.has_been_bootstrapped?

node['private_chef']['bootstrap']['enable']

Its core use is deciding whether or not we run the "bootstrap"
recipe. The bootstrap recipe:

• Possibly installs add_ons
• Creates an RSA key for the pivotal user if one isn't on disk
• Run the CefServerDataBootstrap (which will create pivotal and use
  the key we created)

Additionally it is used to control whether we start oc_id in the
oc_id recipe.

By default (in the attribute file) it is set to true. It can be set
to false in the following cases:

• in recipes/default.rb if either of these checks are true:

if OmnibusHelper.has_been_bootstrapped? or
    BootstrapPreflightValidator.new(node).bypass_bootstrap?
  node.set['private_chef']['bootstrap']['enable'] = false
end

Since this happens before the oc_id recipe is run

• During configuration parsing if we are running on a frontend
• During configuration parsing if we are running on a backend and
  the :bootstrap attribute for our node isn't true

BootstrapPreflightValidator.new(node).bypass_bootstrap?

According to the comment in the definition:

In order to support a stateless standalone that connects to all-external
backend components, allow data bootstrapping to be bypassed when
no chef-server-running.json is present but a secrets file is present.

  def bypass_bootstrap?
    first_run? && secrets_exists? && PrivateChef["use_chef_backend"]
  end

backend_secondary?

Used to guard restarts of services throughout the cookbooks.

is_data_master?

Used throughout the cookbooks to guard service restarts and the
application of schemas.

Defined in enterprise-chef-common in libraries/helper.rb

# Determine if the node is the master for data storage replication
# purposes.
#
# This will return `true` if the node is any of the following:
#
#   * A stand-alone EC install
#   * A tier-topology backend machine
#   * An HA topology backend keepalived master machine
#
# Any other machine will get `false`.
#
# @param node [Chef::Node] node
# @return [Boolean]
def self.is_data_master?(node)
  project_name = node['enterprise']['name']
  topology = node[project_name]['topology']
  role = node[project_name]['role']

  case topology
  when 'standalone'
    true # by definition
  when 'tier'
    role == 'backend'
  when 'ha'
    if role == 'backend'
      dir = node[project_name]['keepalived']['dir']
      cluster_status_file = "#{dir}/current_cluster_status"

      if File.exists?(cluster_status_file)
        File.open(cluster_status_file).read.chomp == 'master'
      else
        # If the file doesn't exist, then we are most likely doing
        # the initial setup, because keepalived must be configured
        # after everything else.  In this case, we'll consider
        # ourself the master if we're defined as the bootstrap
        # server
        is_bootstrap_server?(node)
      end
    else
      false # frontends can't be masters, by definition
    end
  end
end

Start by tracing the bootstrap guards and their uses in libraries/preflight_postgres_validator.rb, libraries/preflight_solr_validator.rb, recipes/bootstrap.rb, recipes/default.rb, and recipes/partybus.rb. Read the referenced BootstrapPreflightValidator and is_data_master? definitions, then map the current paths and state changes. Done would require an agreed, behavior-preserving simplification of bootstrap decisions and corresponding coverage.