Consider changing the Log Cache default host

Issue

CAPI currently retrieves container metric stats from Log Cache. We're in the process of moving Log Cache from the Doppler instance group to its own instance group, which necessitates a cloud_controller property change in order for CAPI to continue to pull container metrics from Log Cache.

Context

Currently, CAPI defines the default Log Cache Host as "doppler.service.cf.internal" in

• capi-release here,
• and, cloud_controller_ng here.

Our PR to move Log Cache to its own instance group also overwrites the default cc.logcache.host property with the new Log Cache host address, "log-cache.service.cf.internal", which should work fine for now.

Eventually we'd expect that the default Log Cache host should be changed to the new host address.

Steps to Reproduce

• Deploy CF from the PR referenced above
  • Do not include the bit that overwrites the cc.logcache.host property
• Push an application

Expected result

Everything works as normal

Current result

• No container metrics available and error message that includes "Stats server temporarily unavailable."
• CAPI logs on the api VM show errors related to connecting to Log Cache

/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:376:in `request_response'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:180:in `block in request_response'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:179:in `request_response'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:171:in `block (3 levels) in rpc_stub_class'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/client.rb:32:in `block in container_metrics'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/client.rb:49:in `with_request_error_handling'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/client.rb:31:in `container_metrics'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/traffic_controller_decorator.rb:49:in `get_container_metrics'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/traffic_controller_decorator.rb:20:in `block in container_metrics'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/traffic_controller_decorator.rb:19:in `loop'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/traffic_controller_decorator.rb:19:in `container_metrics'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/cloud_controller/diego/reporters/instances_stats_reporter.rb:124:in `envelopes'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/cloud_controller/diego/reporters/instances_stats_reporter.rb:23:in `stats_for_app'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/cloud_controller/backends/instances_reporters.rb:25:in `stats_for_app'

Possible Fix

• Change the default in new versions of capi-release and cloud_controller_ng moving forward.
• Remove the overwrite cc.logcache.host in cf-deployment

name of issue screenshot

N/A

Compare the default host in capi-release jobs/cloud_controller_ng/spec and cloud_controller_ng config/cloud_controller.yml, then trace the cc.logcache.host override in cf-deployment. Done means the defaults use log-cache.service.cf.internal, the deployment override can be removed, and container metrics work without the Stats server error.