aws / aws/containers-roadmap

[ECS/Fargate][bug]: Unable to start new tasks on platform 1.4.0 because `ConnotCreateVolumeError`

Open
#2,599 12 comments 1 reaction 1 assignee Claimed by @herrhound View on GitHub
Closing Soon ECS Fargate Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
We have been running a few ECS services on Fargate using platform 1.4.0. All of the services work well with a single container on the task. But when I move a service into multiple containers for the same task, when I have the trigger deployment or force-update-deployment, sometimes it shows the issue.

```
Task is stopping
ConnotCreateVolumeError: unable mirror /var/lib/two/volumes/4961892bc7e04f3183c56118290cb54a/volumes/vf-2902674043 to /var/lib/two/volumes/4961892bc7e04f3183c86118298cb540/volumes/v sns: failed to copy existing contents to task volume unable to copy contents from the image /var/lib/two/volumes/4961892bc7e04f3183c86118290cb540/volumes/v_sns/var/lib/two/volumes/4961892bc7e04f3183c66118290cb540/volumes/vf-2902674043: failed to mkdir /var/lib/two/volumes/4961892bc7e04f3183c86118298cb54a/volumes/v_sns/template: mkdir
/var/lib/two/volumes/4961892bc7e04f3183c86118290c8540/volumes/v sns/template: file exists
```

**Which service(s) is this request for?**
This could be ECS/Fargate

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I have researched and read a lot of the documents about the bind mount, and also double-checked the tutorial configuration from AWS here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bind-mount-examples.html#:~:text=To%20expose%20a%20path%20and%20its%20contents%20in%20a%20Dockerfile%20to%20a%20container

But I think my config does not any the problem. I'm confused a lot because it will deploy successfully after failing many times.

**Are you currently working around this issue?**
How are you currently solving this problem?

**Additional context**
Have added attachments of the task definition, Dockerfile, and some screenshots

**Attachments**

Image

Dockerfile config Volume I have remove somethings which doesn't need

```
ARG GOLANG_URI=golang:1.21.1-alpine
ARG ALPINE_URI=alpine:3.18.5
ARG VERSION
ARG BUILD

FROM $GOLANG_URI AS build

WORKDIR /go/src/github.com/sns/app-data

RUN set -x \
&& apk add --no-cache ca-certificates tzdata util-linux\
&& apk add --no-cache --virtual .build-deps git make

COPY . .

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 VERSION=${VERSION} BUILD=${BUILD} make build-all \
&& go clean --modcache \
&& apk del .build-deps

FROM $ALPINE_URI

ENV RESOURCE_PATH=/sns
ENV PATH="$PATH:${RESOURCE_PATH}/bin"

COPY --from=build /go/bin/ ${RESOURCE_PATH}/bin/
COPY --from=build /go/src/github.com/sns/app-data/template/ ${RESOURCE_PATH}/template/
COPY --from=build /go/src/github.com/sns/app-data/database/ ${RESOURCE_PATH}/database/

VOLUME ["${RESOURCE_PATH}","/tmp"]

```

- TaskDefination

```
{
"taskDefinitionArn": "xx",
"containerDefinitions": [
{
"name": "my-app-cms",
"image": "my-app",
....
"mountPoints": [{
"sourceVolume": "v__var_lib_amazon",
"containerPath": "/var/lib/amazon",
"readOnly": false
},
{
"sourceVolume": "v__var_log_amazon",
"containerPath": "/var/log/amazon",
"readOnly": false
},
{
"sourceVolume": "v__tmp",
"containerPath": "/tmp",
"readOnly": false
},
{
"sourceVolume": "v__sns",
"containerPath": "/sns",
"readOnly": false
}
],
....
},
{
"name": "my-app-scheduler",
"image": "my-app",
....
"mountPoints": [
{
"sourceVolume": "v__var_lib_amazon",
"containerPath": "/var/lib/amazon",
"readOnly": false
},
{
"sourceVolume": "v__var_log_amazon",
"containerPath": "/var/log/amazon",
"readOnly": false
},
{
"sourceVolume": "v__tmp",
"containerPath": "/tmp",
"readOnly": false
},
{
"sourceVolume": "v__sns",
"containerPath": "/sns",
"readOnly": false
}
],
....
}
],
...
"volumes": [
{
"name": "v__var_lib_amazon",
"host": {}
},
{
"name": "v__sns",
"host": {}
},
{
"name": "v__tmp",
"host": {}
},
{
"name": "v__var_log_amazon",
"host": {}
}
],
....
"placementConstraints": [],
"compatibilities": [
"EC2",
"FARGATE"
],
"requiresCompatibilities": [
"FARGATE"
],
....
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.