serverless / serverless/serverless
'Concurrent update operation' error for multi-function service results in both deployment and rollback failure.
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 46.9k
- Forks
- 5.7k
- Avg merge
- 10h 7m
- Merged PRs (30d)
- 57
Description
This is a Bug Report
Description
For bug reports:
-
What went wrong?
Serverless deploy failed due to a concurrent update operation on my multi-function (7 functions) java8 service. Subsequently, the Rollback also fails. I have to manually do a "User Initiated" rollback to get it to rollback completely.
This has never happened before upgrading from serverless 1.25.0 to serverless 1.27.x . -
What did you expect should have happened?
Serverless deploy success for my multi-function (7 functions) java8 service. I was able to reproduce 100% success with serverless 1.25.0 and 100% error/failure with serverless 1.27.x (tried with both 1.27.0 and 1.27.2). It would also be nice to not have to manually initiate a rollback to complete the failed rollback. -
What was the config you used?
I am using Maven to package the code for each function. There are 7 functions defined in my serverless.yaml .
service: MyLambdaService
plugins:
- serverless-plugin-log-retention
provider:
name: aws
runtime: java8
stage: ${opt:stage}
region: ${opt:region}
memorySize: 256
timeout: 300
schedule: 1 hour
role: myLambdaRole
vpc:
securityGroupIds:
- Ref: myLambdaSG
subnetIds:
Fn::Split:
- ', '
- Fn::ImportValue:
Fn::Sub: ${opt:stage}-NATSubnetsList
environment:
DB_PORT: 5432
DB_HOST: ${opt:dbhost}
DB_USERNAME: ${opt:dbuser}
DB_PASSWORD: ${opt:dbpassword}
DB_NAME: mydb
KINESIS_REGION: ${self:provider.region}
REDIS_URL:
Fn::ImportValue: MyRedis-${opt:stage}-JobUrl
custom:
logRetentionInDays: 5
enabled:
prod: true
other: false
package:
individually: true
- What stacktrace or error message from your provider did you see?
Both the stacktrace and the provider error message are located below in "Additional Data"
For feature proposals:
- What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
- If there is additional config how would it look
Similar or dependent issues:
- None
Additional Data
- Serverless Framework Version you're using: 1.27.x
- Operating System: macos, debian
- Stack Trace:
Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (16.28 MB)...
Serverless: Uploading service .zip file to S3 (15.33 MB)...
Serverless: Uploading service .zip file to S3 (15.37 MB)...
Serverless: Uploading service .zip file to S3 (29.21 MB)...
Serverless: Uploading service .zip file to S3 (29.92 MB)...
Serverless: Uploading service .zip file to S3 (29.74 MB)...
Serverless: Uploading service .zip file to S3 (10.4 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
............................
Serverless: Operation failed!
Serverless Error ---------------------------------------
An error occurred: ManageLambdaFunction - The function could not be updated due to a concurrent update operation..
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com
Your Environment Information -----------------------------
OS: linux
Node Version: 8.11.1
Serverless Version: 1.27.0
- Provider Error messages:
03:32:02 UTC+0000 UPDATE_ROLLBACK_FAILED AWS::CloudFormation::Stack MyLambda-Automation-qa The following resource(s) failed to update: [PrepareManageLambdaFunction, PublishOptimizationLambdaFunction, ManageLambdaFunction, CoordinateLambdaFunction, TrainLambdaFunction, OptimizeLambdaFunction].
03:32:01 UTC+0000 UPDATE_FAILED AWS::Lambda::Function PrepareManageLambdaFunction The function could not be updated due to a concurrent update operation.
03:32:01 UTC+0000 UPDATE_FAILED AWS::Lambda::Function PublishOptimizationLambdaFunction The function could not be updated due to a concurrent update operation.
03:32:01 UTC+0000 UPDATE_FAILED AWS::Lambda::Function OptimizeLambdaFunction The function could not be updated due to a concurrent update operation.
03:32:00 UTC+0000 UPDATE_FAILED AWS::Lambda::Function CoordinateLambdaFunction The function could not be updated due to a concurrent update operation.
03:32:00 UTC+0000 UPDATE_FAILED AWS::Lambda::Function ManageLambdaFunction The function could not be updated due to a concurrent update operation.
03:32:00 UTC+0000 UPDATE_FAILED AWS::Lambda::Function TrainLambdaFunction The function could not be updated due to a concurrent update operation.
03:31:59 UTC+0000 UPDATE_IN_PROGRESS AWS::Lambda::Function OptimizeLambdaFunction
03:31:59 UTC+0000 UPDATE_IN_PROGRESS AWS::Lambda::Function ManageLambdaFunction
03:31:59 UTC+0000 UPDATE_IN_PROGRESS AWS::Lambda::Function PrepareManageLambdaFunction
03:31:59 UTC+0000 UPDATE_IN_PROGRESS AWS::Lambda::Function PublishOptimizationLambdaFunction
03:31:59 UTC+0000 UPDATE_IN_PROGRESS AWS::Lambda::Function CoordinateLambdaFunction
03:31:58 UTC+0000 UPDATE_IN_PROGRESS AWS::Lambda::Function TrainLambdaFunction
03:31:57 UTC+0000 UPDATE_COMPLETE AWS::S3::Bucket ServerlessDeploymentBucket
03:31:45 UTC+0000 UPDATE_COMPLETE AWS::Logs::LogGroup TrainLogGroup
03:31:43 UTC+0000 UPDATE_IN_PROGRESS AWS::Logs::LogGroup TrainLogGroup
03:31:37 UTC+0000 UPDATE_COMPLETE AWS::Logs::LogGroup CoordinateLogGroup
03:31:37 UTC+0000 UPDATE_COMPLETE AWS::Logs::LogGroup OptimizeLogGroup
03:31:37 UTC+0000 UPDATE_COMPLETE AWS::Logs::LogGroup PrepareManageLogGroup
03:31:37 UTC+0000 UPDATE_COMPLETE AWS::Logs::LogGroup PublishOptimizationLogGroup
03:31:37 UTC+0000 UPDATE_COMPLETE AWS::Logs::LogGroup PublishTrainingLogGroup
03:31:37 UTC+0000 UPDATE_COMPLETE AWS::Logs::LogGroup ManageLogGroup
03:31:37 UTC+0000 UPDATE_IN_PROGRESS AWS::Logs::LogGroup CoordinateLogGroup
03:31:37 UTC+0000 UPDATE_IN_PROGRESS AWS::Logs::LogGroup PublishOptimizationLogGroup
03:31:37 UTC+0000 UPDATE_IN_PROGRESS AWS::Logs::LogGroup OptimizeLogGroup
03:31:37 UTC+0000 UPDATE_IN_PROGRESS AWS::Logs::LogGroup PrepareManageLogGroup
03:31:37 UTC+0000 UPDATE_IN_PROGRESS AWS::Logs::LogGroup PublishTrainingLogGroup
03:31:37 UTC+0000 UPDATE_IN_PROGRESS AWS::S3::Bucket ServerlessDeploymentBucket
03:31:37 UTC+0000 UPDATE_IN_PROGRESS AWS::Logs::LogGroup ManageLogGroup
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with the seven-function Java 8 service configuration and inspect the CloudFormation update and rollback events for AWS Lambda functions. Compare deployment behavior between Serverless Framework 1.25.0 and 1.27.x. Done means the multi-function deployment succeeds without concurrent-update errors and a failed deployment can roll back completely without manual intervention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, javascript
- Domain
- backend, cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100