podman-docker integration
- Dominant language
- Go
- Stars
- 329
- Forks
- 52
- PR merge metrics
- No merged PRs in 30d
Description
This issue seems to be similar to https://github.com/k14s/kbld/issues/19 but the setup is different.
Have a typical setup for ``kbld`` where it is doing image build and pushing image to remote registry. When it fills out the image digest reference in the deployment, it is using the digest which matched what was stored in local image build cache. The digest which the image was stored under in the remote image registry is different though, meaning the image cannot be found by the deployment.
Final step of ``kbld`` outputs:
```
resolve | final: custom-jupyterhub -> lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-
jupyterhub@sha256:0fc073a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6
```
Inspection of the image in the local build cache yields:
```
[
{
"Id": "769906fbf88ba54ba5a5267749edda68c2aea96a630b2e2c9aee3b7042eaa860",
"Digest": "sha256:0fc073a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6",
"RepoTags": [
"localhost/kbld:custom-jupyterhub-769906fbf88ba54ba5a5267749edda68c2aea96a630b2e2c9aee3b7042eaa8
60",
"lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub:kbld-rand-1586
680932345334383-93222136210142"
],
"RepoDigests": [
"lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub@sha256:0fc073a
041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6",
"localhost/kbld@sha256:0fc073a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6"
],
```
The image reference in the deployment is:
```
metadata:
annotations:
kbld.k14s.io/images: |
- Metas:
- Path: /home/eduk8s/hub-v3
Type: local
URL: lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub@sha256:0fc073a
041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6
...
spec:
containers:
- image: lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub@sha256:0fc07
3a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6
```
The error from ``kapp`` when deploying is:
```
8:47:22AM: ^ Pending: ErrImagePull (message: rpc error: code = Unknown desc = Error response from daemon
: manifest for lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub@sha256:0fc07
3a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6 not found: manifest unknown: manifest unknown)
```
If you rerun just ``kbld`` with nothing changed, you get the error:
```
kbld: Error:
- Resolving image 'custom-jupyterhub': Expected to find same repo digest, but found []string{"lab-jupyter-on
-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub@sha256:0fc073a041e341ca3db2fe40a570d5c22e
b529b591fc90a490737b4d48ca53d6", "lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupy
terhub@sha256:484209beb53e3dae50fc828fa215ac596f9f6f35d6398c113b7ce45b59dec5d2", "localhost/kbld@sha256:0fc0
73a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6", "localhost/kbld@sha256:484209beb53e3dae50fc82
8fa215ac596f9f6f35d6398c113b7ce45b59dec5d2"}
```
which also appears to be picking up the mismatch.
As far as the setup goes, the builds are being done with ``podman`` using the ``podman-docker`` wrapper on Fedora which results in ``podman`` being invoked when ``docker`` is run. Thus ``podman`` is used to push the image to the remote registry. The remote registry self hosted latest version of the standard Docker image registry.
The image registry was empty before starting. After ``kbld`` run, have:
```
$ curl -u xxx:xxx -X GET https://$REGISTRY_HOST/v2/_catalog
{"repositories":["custom-jupyterhub"]}
$ curl -u xxx:xxx -X GET https://$REGISTRY_HOST/v2/custom-jupyterhub/tags/list
{"name":"custom-jupyterhub","tags":["kbld-rand-1586680932345334383-93222136210142"]}
```
Running ``skopeo`` to inspect the remote image get:
```
"Name": "lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub",
"Digest": "sha256:cdaeaddf70987d0b895efccaa735ea6efcdb00228a0bdd9f586c78f663c42b3c",
"RepoTags": [
"kbld-rand-1586680932345334383-93222136210142"
],
...
"Layers": [
"sha256:462816986e4db9b3ba3ff89114709053ed939e75339c1bf64bb6c8c20bb0ac09",
"sha256:8a14c0ec1aad906c9703b3be9a4807671a6367d6496d1941ff4a40f488887816",
"sha256:ea892e6e4eade778ab7349dafb7a078676f4367c7bed5067c2456c204705aeaa",
"sha256:a8abb9d778483d211484c35e61002e6dde8ca8755c0577a82fe265c65590a786",
"sha256:f5266c3eb07a9a052ccde05abe2bd4e763a8ffe316dcaef1c3f1b2935515ec14",
"sha256:255f20c7a19d5ec58bc992bcf3c56cd58c6cd9ffd144c3cbd854a5e4efc72a94",
"sha256:3699fa7667edd14c6e99940406dd9fb4f2d33e0bced900623bd973bd822ab4d5",
"sha256:cf5bbb6d527ad8bc3eb29a4b73fa7acfc1fcb2d4bbf17df1b3d5e4176aad33bd",
"sha256:021fc1973d41319bdb76e723c4f89424d5a9c7a6d3bd5b0d79560722a70e53e3",
"sha256:b971891c534ff0a70e03c988a9e254b6e33d192e61d1e1352061edb69bba7469",
"sha256:5fd45b49b7e12c875ca5110ea78b8ff04ac90c79db1c0bbced7a13d55d6f820a",
"sha256:f291ee79eb30aedf1cf37386b3c3233e193c3a95255a8e8b55e9d5ff41e07409",
"sha256:028d992643957f2d6b967697e91e7c986eb919016aa88fd997e8b91237cb5fed",
"sha256:716df0b1c063d005b933667274c2edd3a0d6bf5405ac1390812066543c812a36",
"sha256:68d6bea7dd2ed44cb44cf70365c75f69e6b30877332dac5d08b333c4de56c6b1"
],
```
So looks like ``kbld`` should be working out what the digest is from the image on the remote image registry after it has been pushed, rather than assuming the image will have the same digest value.
cc @jorgemoralespou
Contributor guide
Research direction
Start by tracing kbld's image push and digest-resolution flow, reproducing the issue with the podman-docker wrapper and a Docker registry. The work is done when the deployment uses the digest reported by the remote registry and rerunning kbld no longer reports a repository-digest mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, kubernetes
- Domain
- cli, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100