abiosoft / abiosoft/colima

Trying to start a project in DDEV but getting an API error (500)

Open
#883 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.8k
Forks
613
PR merge metrics
No merged PRs in 30d

Description

### Description

I ran into something odd. I had a project running with ddev and colima 0.6.2 (on a m1pro with sonoma and vz/virtiofs). i wanted to start a second project in the terminal within vscode. i got the following output:

```
$> ddev start
Starting modi...
Failed to RunSimpleContainer to inspect database version/type: failed to create/start Docker container ({GetExistingDBType-modi-MxdvNY 0x14000220960 0x14000080480 }):API error (500): mkdir /var/lib/docker/overlay2/a784aef8fbf3af0f2a38904f816e31b88e268056b829a6b35028d77864d4a4c2-init: read-only file system, output=
```
when i checked the containers:
```
$> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
11c62cb4fdd6 ddev/ddev-traefik-router:v1.22.4 "/entrypoint.sh --co…" About an hour ago Up About an hour (healthy) 127.0.0.1:80->80/tcp, 127.0.0.1:443->443/tcp, 127.0.0.1:8025-8026->8025-8026/tcp, 127.0.0.1:8036-8037->8036-8037/tcp, 127.0.0.1:10999->10999/tcp ddev-router
afa5b0a2bd88 phpmyadmin:5 "/docker-entrypoint.…" About an hour ago Up About an hour 80/tcp ddev-tests-phpmyadmin
6eeda14e67e0 ddev/ddev-dbserver-mariadb-10.5:v1.22.4-tests-built "/docker-entrypoint.…" About an hour ago Up About an hour (healthy) 127.0.0.1:32777->3306/tcp ddev-tests-db
120c00fe7e45 ddev/ddev-webserver:20231108_new_lagoon-tests-built "/pre-start.sh" About an hour ago Up About an hour (healthy) 8025/tcp, 127.0.0.1:32776->80/tcp, 127.0.0.1:32775->443/tcp ddev-tests-web
c71301f7966d ddev/ddev-ssh-agent:v1.22.4-built "/entry.sh ssh-agent" About an hour ago Up About an hour (healthy) ddev-ssh-agent```

the container listed on the failed startup for the project modi doesnt exist. i then tried to poweroff ddev

```
$> ddev poweroff
Failed removing custom certs/traefik configuration: failed to create/start Docker container ({poweroff-tHzHOH 0x1400021e1e0 0x1400020c480 }):API error (500): mkdir /var/lib/docker/overlay2/849ce4583dc2cf404267058b8b8952b57615930fd513a63748ba6eb4bdd1ace1-init: read-only file system
Container ddev-tests-phpmyadmin Stopped
Container ddev-tests-web Stopped
Error response from daemon: container afa5b0a2bd88aac381e507e835604d2282c4fce285ee43807c4c295c4841fc79: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3ab71b22bcec4d2a8dcdd782d71ef9981c9931f87f8e22aaf499c48e8b57a2fb: read-only file system
Failed to docker-compose down: composeCmd failed to run 'COMPOSE_PROJECT_NAME=ddev-tests docker-compose -f /Users/rkoller/Sites/tests/.ddev/.ddev-docker-compose-full.yaml down', action='[]', err='exit status 1', stdout='', stderr=' Container ddev-tests-phpmyadmin Stopping
Container ddev-tests-web Stopping
Container ddev-tests-phpmyadmin Stopped
Container ddev-tests-phpmyadmin Removing
Container ddev-tests-phpmyadmin Error while Removing
Container ddev-tests-web Stopped
Container ddev-tests-web Removing
Container ddev-tests-web Error while Removing
Error response from daemon: container afa5b0a2bd88aac381e507e835604d2282c4fce285ee43807c4c295c4841fc79: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3ab71b22bcec4d2a8dcdd782d71ef9981c9931f87f8e22aaf499c48e8b57a2fb: read-only file system'
Removing container: ddev-tests-phpmyadmin
Failed to stop project tests:
could not remove container ddev-tests-phpmyadmin: API error (500): container afa5b0a2bd88aac381e507e835604d2282c4fce285ee43807c4c295c4841fc79: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3ab71b22bcec4d2a8dcdd782d71ef9981c9931f87f8e22aaf499c48e8b57a2fb: read-only file system
```

i then tried a hard reset:
```
$> docker rmi -f $(docker images -q)
Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3892998772: read-only file system
Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json1151577697: read-only file system
Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3079715760: read-only file system
Error response from daemon: conflict: unable to delete 1066e79ef629 (cannot be forced) - image is being used by running container 6eeda14e67e0
Error response from daemon: conflict: unable to delete a8483ea26b94 (cannot be forced) - image is being used by running container c71301f7966d
Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json1453494246: read-only file system
Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3496979304: read-only file system
Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3644803181: read-only file system
Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3927001417: read-only file system
Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json2578752554: read-only file system
Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3753705229: read-only file system
Error response from daemon: conflict: unable to delete 97c388ff3350 (cannot be forced) - image is being used by running container 11c62cb4fdd6
Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json389738583: read-only file system
```
and
```
$> docker rm -f $(docker ps -aq)
Error response from daemon: Could not kill running container 11c62cb4fdd60e4793b2ce078dec0633f1134a1487f06e6009018caff80c1952, cannot remove - container 11c62cb4fdd6 PID 284811 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes
Error response from daemon: container afa5b0a2bd88aac381e507e835604d2282c4fce285ee43807c4c295c4841fc79: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3ab71b22bcec4d2a8dcdd782d71ef9981c9931f87f8e22aaf499c48e8b57a2fb: read-only file system
Error response from daemon: Could not kill running container 6eeda14e67e032c9f59bba1c64a23cb80d17a4a082d65898113449619b8527cb, cannot remove - container 6eeda14e67e0 PID 280580 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes
Error response from daemon: container 120c00fe7e45844a8c7860ce9d76ed41e891d4b7344665dbe8d466a99f472802: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/391d9432305b168f4a622f4927d74b55f046191a65201f2d27bbba67516fd78d: read-only file system
Error response from daemon: container c71301f7966dfeaacda4248f1ce4dd2a98b0fe61d56fdae435ea531202bd0fca: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/d8fd6e612336f8b08c40b77cfa3afa9fbb49c3b6f1a5cd2cf756e64032d5750f: read-only file system```

i also tried to execute `docker rm -f $(docker ps -aq)`inside the colima vm after `colima ssh` same result like on the host.

no idea what caused it and or if you need any further informations?

### Version

```
$> colima version && limactl --version && qemu-img --version
colima version 0.6.2
git commit: 22d7e5fbc86d5b8e3b27065a762800bc7960a0ff

runtime: docker
arch: aarch64
client: v24.0.7
server: v24.0.5
limactl version 0.18.0
qemu-img version 8.1.2
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
ddev version v1.22.4-37-gca6b43689
```

### Operating System

- [ ] macOS Intel <= 13 (Ventura)
- [ ] macOS Intel >= 14 (Sonoma)
- [ ] Apple Silicon <= 13 (Ventura)
- [X] Apple Silicon >= 14 (Sonoma)
- [ ] Linux

### Output of `colima status`

```
$> colima status
INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/rkoller/.colima/default/docker.sock
```

### Reproduction Steps

I've outlined the steps that snowballed into the trouble. not sure if it is reproducible.

### Expected behaviour

the error cascade happened out of the blue.

### Additional context

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.