slimtoolkit / slimtoolkit/slim

docker-slim build causes "Could not find a suitable TLS CA certificate bundle" in Python requests

Open
#101 14 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
23.4k
Forks
840
No merged PRs in 30d

Description

I know nothing about docker-slim, I don't have a clue how it works, but it looks exciting so I'm giving it a try on our Dockerfile for ubuntu.com.

I build the original image as follows:

$ git clone git@github.com:canonical-web-and-design/ubuntu.com.git
...
$ cd ubuntu.com
$ DOCKER_BUILDKIT=1 docker build --tag ubuntu-com .
...
$ docker images ubuntu-com:latest
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu-com          latest              0162e0fc46e5        21 seconds ago      253MB

Then I run docker-slim, which appears to succed, and does indeed more than halve the size of the image:

$ ~/Downloads/dist_linux/docker-slim build --expose 80 ubuntu-com
docker-slim[build]: info=http.probe message='using default probe'
docker-slim[build]: state=started
docker-slim[build]: info=params target=ubuntu-com continue.mode=probe
docker-slim[build]: state=image.inspection.start
docker-slim[build]: info=image id=sha256:0162e0fc46e559ff9b8b12dd57b4ea5d28670b0e388fcc234fee158255f20ee9 size.bytes=253235824 size.human=253 MB
docker-slim[build]: info=image.stack index=0 name='ubuntu-com:latest' id='sha256:0162e0fc46e559ff9b8b12dd57b4ea5d28670b0e388fcc234fee158255f20ee9'
docker-slim[build]: state=image.inspection.done
docker-slim[build]: state=container.inspection.start
docker-slim[build]: info=container status=created name=dockerslimk_2276_20191211104333 id=130d60dacd6dc660602af00a0014af97fa60b7974b75046ad198a2e5191ecca0
docker-slim[build]: info=cmd.startmonitor status=sent
docker-slim[build]: info=event.startmonitor.done status=received
docker-slim[build]: info=container name=dockerslimk_2276_20191211104333 id=130d60dacd6dc660602af00a0014af97fa60b7974b75046ad198a2e5191ecca0 target.port.list=[32774] target.port.info=[80/tcp => 0.0.0.0:32774] message='YOU CAN USE THESE PORTS TO INTERACT WITH THE CONTAINER'
docker-slim[build]: state=http.probe.starting message='WAIT FOR HTTP PROBE TO FINISH'
docker-slim[build]: info=continue.after mode=probe message='no input required, execution will resume when HTTP probing is completed'
docker-slim[build]: info=prompt message='waiting for the HTTP probe to finish'
docker-slim[build]: state=http.probe.running
docker-slim[build]: info=http.probe.ports count=1 targets='32774'
docker-slim[build]: info=http.probe.commands count=1 commands='GET /'
docker-slim[build]: info=http.probe.call status=200 method=GET target=http://127.0.0.1:32774/ attempt=1  time=2019-12-11T10:43:46Z
docker-slim[build]: info=http.probe.summary total=1 failures=0 successful=1
docker-slim[build]: state=http.probe.done 
docker-slim[build]: info=event message='HTTP probe is done'
docker-slim[build]: state=container.inspection.finishing
docker-slim[build]: state=container.inspection.artifact.processing
docker-slim[build]: state=container.inspection.done
docker-slim[build]: state=building message='building minified image'
docker-slim[build]: state=completed
docker-slim[build]: info=results status='MINIFIED BY 2.29X [253235824 (253 MB) => 110452274 (110 MB)]'
docker-slim[build]: info=results  image.name=ubuntu-com.slim image.size='110 MB' data=true
docker-slim[build]: info=results  artifacts.location='/home/robin/Downloads/dist_linux/.docker-slim-state/images/0162e0fc46e559ff9b8b12dd57b4ea5d28670b0e388fcc234fee158255f20ee9/artifacts'
docker-slim[build]: info=results  artifacts.report=creport.json
docker-slim[build]: info=results  artifacts.dockerfile.original=Dockerfile.fat
docker-slim[build]: info=results  artifacts.dockerfile.new=Dockerfile
docker-slim[build]: info=results  artifacts.seccomp=ubuntu-com-seccomp.json
docker-slim[build]: info=results  artifacts.apparmor=ubuntu-com-apparmor-profile
docker-slim[build]: state=done
docker-slim[build]: info=report file='slim.report.json'

$ docker images ubuntu-com.slim:latest
REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE
ubuntu-com.slim     latest              834c263189e7        About a minute ago   110MB

But now if I run the site from the new image:

$ docker run -ti -p 8222:80 ubuntu-com.slim:latest
2019-12-11 10:37:03.147Z INFO talisker.sentry "Raven is not configured (logging is disabled). Please see the documentation for more information."
2019-12-11 10:37:03.184Z INFO gunicorn.error "Starting gunicorn 19.10.0"                                                                                                                                          
2019-12-11 10:37:03.185Z INFO gunicorn.error "Listening at: http://0.0.0.0:80 (7)"                                                                                                                                 
2019-12-11 10:37:03.185Z INFO gunicorn.error "Using worker: sync"                                  
2019-12-11 10:37:03.187Z INFO gunicorn.error "Booting worker with pid: 11"                                                                                                                                         
2019-12-11 10:37:03.239Z INFO gunicorn.error "Booting worker with pid: 12"                                                                                                                                         
2019-12-11 10:37:03.252Z INFO gunicorn.error "Booting worker with pid: 13"                                                                                                                                         
2019-12-11 10:37:03.284Z INFO gunicorn.error "Booting worker with pid: 14"                                                                                                                                         
2019-12-11 10:37:03.378Z INFO gunicorn.error "Booting worker with pid: 15"                                                                                                                                         
2019-12-11 10:37:04.262Z INFO talisker.flask "updating raven config from flask app"                                       
2019-12-11 10:37:04.263Z INFO talisker.sentry "Raven is not configured (logging is disabled). Please see the documentation for more information."                                                                  
2019-12-11 10:37:04.282Z INFO talisker.flask "updating raven config from flask app"                                                    
2019-12-11 10:37:04.283Z INFO talisker.sentry "Raven is not configured (logging is disabled). Please see the documentation for more information."
2019-12-11 10:37:04.379Z INFO talisker.flask "updating raven config from flask app"                
2019-12-11 10:37:04.379Z INFO talisker.sentry "Raven is not configured (logging is disabled). Please see the documentation for more information."
2019-12-11 10:37:04.494Z INFO talisker.flask "updating raven config from flask app"         
2019-12-11 10:37:04.494Z INFO talisker.sentry "Raven is not configured (logging is disabled). Please see the documentation for more information."
2019-12-11 10:37:04.511Z INFO talisker.flask "updating raven config from flask app"                
2019-12-11 10:37:04.512Z INFO talisker.sentry "Raven is not configured (logging is disabled). Please see the documentation for more information."

Then I browse to http://127.0.0.1:8222/blog, the "blog" feed fails to load, and I see these errors in the image output:

2019-12-11 10:37:10.861Z ERROR talisker.requests "http request failure" url=https://admin.insights.ubuntu.com/wp-json/wp/v2/posts? qs="?per_page=<len 1>&page=<len 1>&tags_exclude=<len 14>&sticky=<len 4>&_embed=<len 4>" qs_size=73 method=GET host=admin.insights.ubuntu.com service=ubuntu.com request_id=adacabfb-fd60-4
368-aa23-48f7e6de26f9
Traceback (most recent call last):
  File "/root/.local/lib/python3.6/site-packages/talisker/requests.py", line 173, in send
    return func(request, **kwargs)
  File "/root/.local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/root/.local/lib/python3.6/site-packages/requests/adapters.py", line 416, in send
    self.cert_verify(conn, request.url, verify, cert)
  File "/root/.local/lib/python3.6/site-packages/requests/adapters.py", line 228, in cert_verify
    "invalid path: {}".format(cert_loc))
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /root/.local/lib/python3.6/site-packages/certifi/cacert.pem
2019-12-11 10:37:10.864Z ERROR flask.app "Exception on /blog/latest-news [GET]" service=ubuntu.com request_id=adacabfb-fd60-4368-aa23-48f7e6de26f9
Traceback (most recent call last):
  File "/root/.local/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/root/.local/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/root/.local/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/root/.local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/root/.local/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/root/.local/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/root/.local/lib/python3.6/site-packages/canonicalwebteam/blog/flask.py", line 63, in latest_news
    limit=flask.request.args.get("limit", "3"),
  File "/root/.local/lib/python3.6/site-packages/canonicalwebteam/blog/common_view_logic.py", line 268, in get_latest_news
    sticky=True,
  File "/root/.local/lib/python3.6/site-packages/canonicalwebteam/blog/wordpress_api.py", line 89, in get_articles
    response = api_session.get(url)
  File "/root/.local/lib/python3.6/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/root/.local/lib/python3.6/site-packages/talisker/requests.py", line 190, in request
    return func(method, url, **kwargs)
  File "/root/.local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/.local/lib/python3.6/site-packages/talisker/requests.py", line 173, in send
    return func(request, **kwargs)
  File "/root/.local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/root/.local/lib/python3.6/site-packages/requests/adapters.py", line 416, in send
    self.cert_verify(conn, request.url, verify, cert)
  File "/root/.local/lib/python3.6/site-packages/requests/adapters.py", line 228, in cert_verify
    "invalid path: {}".format(cert_loc))
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /root/.local/lib/python3.6/site-packages/certifi/cacert.pem

The container is using requests to query a wordpress API at https://admin.insights.ubuntu.com/wp-json/wp/v2/posts, and it works in the original ubuntu-com:latest image. Something about the slimming process appears to be removing something the Requests library needs to verify HTTPS certificates, or something.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the documented docker-slim build commands and compare the original and slim images. Start with the failing requests path and /root/.local/lib/python3.6/site-packages/certifi/cacert.pem; trace what the slimming process removes. Done means the slim image can load the blog feed over HTTPS without the CA-bundle error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
devops, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.