cookbook uploads fail when using a https only LB forwarding as http to nginx

Chef Server Version

12.18.14

Platform Details

This was discovered in the Chef Server container solution where the Chef Server habitat pkgs were exported as docker containers.

Configuration

client <== HTTPS ==> F5 LB (https://chef-server.my.org) <== HTTP ==> Chef Server nginx

Scenario:

knife cookbook upload

Steps to Reproduce:

Configure a LB as stated above with only HTTPS listener/forwarder for the VIP but talking HTTP (non-ssl) to the (backend) Chef Server on 8080.

Expected Result:

cookbook upload succeeds

Actual Result:

upload fails with obtuse ridley error.

Analysis:

The ridley error is a result of a protocol mismatch. When the client issues the POST, nginx responds with a 201 and the following:

TRACE: ---- HTTP Status and Header Data: ----
TRACE: HTTP 1.1 201 Created
TRACE: server: openresty/1.11.2.5
TRACE: date: Mon, 07 Jan 2019 15:23:15 GMT
TRACE: content-type: application/json
TRACE: content-length: 1243
TRACE: connection: close
TRACE: x-ops-server-api-version: {"min_version":"0","max_version":"2","request_version":"2","response_version":"2"}
TRACE: x-ops-api-info: flavor=cs;version=12.0.0;oc_erchef=12.18.14
TRACE: location: http://xx.yy.com:443/organizations/xxx/sandboxes/69a0efab73cf16b7b71e5b814e7728a8
TRACE: x-ms-unique-id: m4NPdwRADz4aNHD9XlX
TRACE: ---- End HTTP Status/Header Data ----

Notice the uri scheme and port: location: http://xx.yy.com:443/organizations/xxx/sandboxes/69a0efab73cf16b7b71e5b814e7728a8

The response causes the client to attempt to speak HTTP to the LB SSL port, and results in a termination of the transaction.

This appears to be a result of these lines: https://github.com/chef/chef-server/blob/master/src/nginx/habitat/config/chef_http_lb_common#L38-L42

Nginx needs to build the uri maintaining the proto scheme and port of the original client request as it came into the LB.

Workaround

Configure the F5 to forward all traffic to the HTTPS nginx listener (8443), not the non-SSL (8080) nginx listener. This may require a irule rewrite and it will require the LB to do SSL exchange (with no certification validation) to the nginx backend on 8443.

Start with src/nginx/habitat/config/chef_http_lb_common at lines 38-42 and reproduce the HTTPS-to-HTTP load-balancer setup with knife cookbook upload. Verify that the generated Location URI preserves the client-facing HTTPS scheme and port, and that the upload completes successfully.