PID file not removed and checkStalePid kills itself

Bug report

What operating system and version are you using?

$ osqueryi --line "SELECT version, build, platform FROM os_version;"
 version = 10.13.6
   build = 17G65
platform = darwin

What version of osquery are you using?

$ osqueryi --line "SELECT version from osquery_info;"
version = 3.2.6

What steps did you take to reproduce the issue?

We installed osquery on a fresh macOS High Sierra virtual machine, along with a flagfile and certificates containing settings for working with our Kolide Fleet service.

$ cat /etc/osquery/kolide.flags 
--force
--host_identifier=hostname
--verbose=true
--debug
--tls_dump=true

--tls_hostname=[[redacted]]
--tls_server_certs=/etc/osquery/kolide.crt
--enroll_secret_path=/etc/osquery/kolide_secret

--enroll_tls_endpoint=/api/v1/osquery/enroll

--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_tls_refresh=30
--config_tls_max_attempts=3

--disable_distributed=false
--distributed_plugin=tls
--distributed_interval=10
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write

--logger_plugin=tls
--logger_tls_endpoint=/api/v1/osquery/log
--logger_tls_period=10

What did you expect to see?.

We expect to see osquery running as usual when the system has loaded.

What did you see instead?

After installing osquery and rebooting the system two times, osqueryd no longer properly loads on system startup. It will load on occasion, but rarely and unreliably. The following log line appears in /var/log//system.log:

com.apple.xpc.launchd[1] (co.kolide.osquery.enroll[62]): Service exited due to signal: Killed: 9 sent by osqueryd[62]

Additionally, /var/osquery/osqueryd.pidfile is not removed, which is where I believe the problem stems from.

Because osquery loads early in system startup, it seems to consistently get the same process ID (in this case, 62). Reading through osquery/osquery/core/system.cpp, I do not see anywhere that checkStalePid checks to see if the process number in the pid file is actually itself (though I could be missing something.)

From what I can tell, osqueryd does not remove its pid file on shutdown. Then when it loads on startup, it checks to see if a pid file exists, which it does. It then looks to see if the process matching the pid is osqueryd, which it is, because it itself was assigned the same pid as last time. And then it issues target.kill() on itself and dies.

This explains why the problem manifests after the second reboot, because during the initial install it is assigned a high pid number, and after the first reboot that pid file has not been changed yet to reflect the lower pid. It also explains why osqueryd will actually load intermittently, because on occasion the startup services will load in a different order and osqueryd will be assigned a different pid on startup.

Finally, when we replace --force with --ephemeral in our flagfile, everything works as expected, because it is not worrying about pid files.