serverless / serverless/serverless

'Concurrent update operation' error for multi-function service results in both deployment and rollback failure.

Open
#4,964 19 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug/external cat/deployment
Dominant language
JavaScript
Stars
46.9k
Forks
5.7k
Avg merge
10h 7m
Merged PRs (30d)
57

Description

This is a Bug Report

Description

For bug reports:

  • What went wrong?
    Serverless deploy failed due to a concurrent update operation on my multi-function (7 functions) java8 service. Subsequently, the Rollback also fails. I have to manually do a "User Initiated" rollback to get it to rollback completely.
    This has never happened before upgrading from serverless 1.25.0 to serverless 1.27.x .

  • What did you expect should have happened?
    Serverless deploy success for my multi-function (7 functions) java8 service. I was able to reproduce 100% success with serverless 1.25.0 and 100% error/failure with serverless 1.27.x (tried with both 1.27.0 and 1.27.2). It would also be nice to not have to manually initiate a rollback to complete the failed rollback.

  • What was the config you used?
    I am using Maven to package the code for each function. There are 7 functions defined in my serverless.yaml .

service: MyLambdaService

plugins:
  - serverless-plugin-log-retention

provider:
  name: aws
  runtime: java8
  stage: ${opt:stage}
  region: ${opt:region}
  memorySize: 256
  timeout: 300
  schedule: 1 hour
  role: myLambdaRole
  vpc:
      securityGroupIds:
        - Ref: myLambdaSG
      subnetIds:
        Fn::Split:
        - ', '
        - Fn::ImportValue:
            Fn::Sub: ${opt:stage}-NATSubnetsList
  environment:
    DB_PORT: 5432
    DB_HOST: ${opt:dbhost}
    DB_USERNAME: ${opt:dbuser}
    DB_PASSWORD: ${opt:dbpassword}
    DB_NAME: mydb
    KINESIS_REGION: ${self:provider.region}
    REDIS_URL:
      Fn::ImportValue: MyRedis-${opt:stage}-JobUrl

custom:
  logRetentionInDays: 5
  enabled:
    prod: true
    other: false

package:
  individually: true
  • What stacktrace or error message from your provider did you see?
    Both the stacktrace and the provider error message are located below in "Additional Data"

For feature proposals:

  • What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
  • If there is additional config how would it look

Similar or dependent issues:

  • None

Additional Data

  • Serverless Framework Version you're using: 1.27.x
  • Operating System: macos, debian
  • Stack Trace:
Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (16.28 MB)...
Serverless: Uploading service .zip file to S3 (15.33 MB)...
Serverless: Uploading service .zip file to S3 (15.37 MB)...
Serverless: Uploading service .zip file to S3 (29.21 MB)...
Serverless: Uploading service .zip file to S3 (29.92 MB)...
Serverless: Uploading service .zip file to S3 (29.74 MB)...
Serverless: Uploading service .zip file to S3 (10.4 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
............................
Serverless: Operation failed!
 
  Serverless Error ---------------------------------------
 
  An error occurred: ManageLambdaFunction - The function could not be updated due to a concurrent update operation..
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           8.11.1
     Serverless Version:     1.27.0

  • Provider Error messages:
03:32:02 UTC+0000	UPDATE_ROLLBACK_FAILED	AWS::CloudFormation::Stack	MyLambda-Automation-qa	The following resource(s) failed to update: [PrepareManageLambdaFunction, PublishOptimizationLambdaFunction, ManageLambdaFunction, CoordinateLambdaFunction, TrainLambdaFunction, OptimizeLambdaFunction].
03:32:01 UTC+0000	UPDATE_FAILED	AWS::Lambda::Function	PrepareManageLambdaFunction	The function could not be updated due to a concurrent update operation.
03:32:01 UTC+0000	UPDATE_FAILED	AWS::Lambda::Function	PublishOptimizationLambdaFunction	The function could not be updated due to a concurrent update operation.
03:32:01 UTC+0000	UPDATE_FAILED	AWS::Lambda::Function	OptimizeLambdaFunction	The function could not be updated due to a concurrent update operation.
03:32:00 UTC+0000	UPDATE_FAILED	AWS::Lambda::Function	CoordinateLambdaFunction	The function could not be updated due to a concurrent update operation.
03:32:00 UTC+0000	UPDATE_FAILED	AWS::Lambda::Function	ManageLambdaFunction	The function could not be updated due to a concurrent update operation.
03:32:00 UTC+0000	UPDATE_FAILED	AWS::Lambda::Function	TrainLambdaFunction	The function could not be updated due to a concurrent update operation.
03:31:59 UTC+0000	UPDATE_IN_PROGRESS	AWS::Lambda::Function	OptimizeLambdaFunction	
03:31:59 UTC+0000	UPDATE_IN_PROGRESS	AWS::Lambda::Function	ManageLambdaFunction	
03:31:59 UTC+0000	UPDATE_IN_PROGRESS	AWS::Lambda::Function	PrepareManageLambdaFunction	
03:31:59 UTC+0000	UPDATE_IN_PROGRESS	AWS::Lambda::Function	PublishOptimizationLambdaFunction	
03:31:59 UTC+0000	UPDATE_IN_PROGRESS	AWS::Lambda::Function	CoordinateLambdaFunction	
03:31:58 UTC+0000	UPDATE_IN_PROGRESS	AWS::Lambda::Function	TrainLambdaFunction	
03:31:57 UTC+0000	UPDATE_COMPLETE	AWS::S3::Bucket	ServerlessDeploymentBucket	
03:31:45 UTC+0000	UPDATE_COMPLETE	AWS::Logs::LogGroup	TrainLogGroup	
03:31:43 UTC+0000	UPDATE_IN_PROGRESS	AWS::Logs::LogGroup	TrainLogGroup	
03:31:37 UTC+0000	UPDATE_COMPLETE	AWS::Logs::LogGroup	CoordinateLogGroup	
03:31:37 UTC+0000	UPDATE_COMPLETE	AWS::Logs::LogGroup	OptimizeLogGroup	
03:31:37 UTC+0000	UPDATE_COMPLETE	AWS::Logs::LogGroup	PrepareManageLogGroup	
03:31:37 UTC+0000	UPDATE_COMPLETE	AWS::Logs::LogGroup	PublishOptimizationLogGroup	
03:31:37 UTC+0000	UPDATE_COMPLETE	AWS::Logs::LogGroup	PublishTrainingLogGroup	
03:31:37 UTC+0000	UPDATE_COMPLETE	AWS::Logs::LogGroup	ManageLogGroup	
03:31:37 UTC+0000	UPDATE_IN_PROGRESS	AWS::Logs::LogGroup	CoordinateLogGroup
03:31:37 UTC+0000	UPDATE_IN_PROGRESS	AWS::Logs::LogGroup	PublishOptimizationLogGroup	
03:31:37 UTC+0000	UPDATE_IN_PROGRESS	AWS::Logs::LogGroup	OptimizeLogGroup	
03:31:37 UTC+0000	UPDATE_IN_PROGRESS	AWS::Logs::LogGroup	PrepareManageLogGroup	
03:31:37 UTC+0000	UPDATE_IN_PROGRESS	AWS::Logs::LogGroup	PublishTrainingLogGroup	
03:31:37 UTC+0000	UPDATE_IN_PROGRESS	AWS::S3::Bucket	ServerlessDeploymentBucket	
03:31:37 UTC+0000	UPDATE_IN_PROGRESS	AWS::Logs::LogGroup	ManageLogGroup

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the seven-function Java 8 service configuration and inspect the CloudFormation update and rollback events for AWS Lambda functions. Compare deployment behavior between Serverless Framework 1.25.0 and 1.27.x. Done means the multi-function deployment succeeds without concurrent-update errors and a failed deployment can roll back completely without manual intervention.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, javascript
Domain
backend, cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.