Container Image Deploy: "toomanyrequests: Rate exceeded"

Are you certain it's a bug?

• Yes, it looks like a bug

Is the issue caused by a plugin?

• It is not a plugin issue

Are you using the latest v3 release?

• Yes, I'm using the latest v3 release

Is there an existing issue for this?

• I have searched existing issues, it hasn't been reported yet

Issue description

Occasionally, perhaps once every 50 deploys, I receive an error saying "toomanyrequests: Rate exceeded". There is very little else we do with containers or ECR. This is for a project that is using a Container Image runtime and therefore interacting with ECR. I believe the error is from ECR, but am not sure. Is some serverless framework AWS call missing a backoff/retry?

The DockerFile:

FROM public.ecr.aws/shelf/lambda-libreoffice-base:7.4-node16-x86_64 as builder
WORKDIR /usr/app
COPY . ./
RUN npm i
RUN npm run build

FROM public.ecr.aws/shelf/lambda-libreoffice-base:7.4-node16-x86_64
WORKDIR ${LAMBDA_TASK_ROOT}
COPY --from=builder /usr/app/ ./
RUN mkdir -p ./.build/src/templates
COPY --from=builder /usr/app/src/templates/ ./.build/src/templates

Service configuration (serverless.yml) content

service: platform-async-pdf-svc

frameworkVersion: '3'

provider:
  deploymentBucket:
    name: esub-${sls:stage}-deployment-bucket
    serverSideEncryption: AES256
  ecr:
    images:
      docx_to_pdf_image:
        path: ./
  environment: N/A
  logRetentionInDays: 7
  name: aws
  region: ${opt:region, 'us-west-2'}
  runtime: nodejs16.x
  stage: ${opt:stage, 'devtest'}

custom:
  alertTopicNameARN: arn:aws:sns:${self:provider.region}:${AWS::AccountId}:${self:service}-${sls:stage}-alerts-alarm
  bucketName: platform-async-pdf-svc-${sls:stage}-us-west-2-files
  cfPublicKey: N/A
  cfPublicKeyId: N/A
  distributionDomainName: N/A
  esbuild:
    plugins: common/esbuildPlugins/plugins.js
  logRetentionInDays: 30
  tableName: ${self:service}-${sls:stage}

package:
  individually: true
  patterns:
    - src/templates/*.docx
    - '!test/**'
    - '!sdk/**'

plugins:
  - serverless-esbuild
  - serverless-deployment-bucket
  - serverless-iam-roles-per-function
  - serverless-prune-plugin
  - serverless-step-functions

functions:
  createPdfJob:
    handler: src/handlers/createJob.default
    memorySize: 512
    events:
      - httpApi:
          method: POST
          path: /pdf-job
          authorizer:
            type: aws_iam
    iamRoleStatements: N/A
      
  getPdfJob:
    handler: src/handlers/getJob.default
    memorySize: 512
    events:
      - httpApi:
          method: GET
          path: /pdf-job/{jobId}
          authorizer:
            type: aws_iam
    iamRoleStatements: N/A

  ## State Machine Functions
  setStatusInProgress:
    handler: src/handlers/setStatusInProgress.default
    iamRoleStatements: N/A

  setStatusSucceeded:
    handler: src/handlers/setStatusSucceeded.default
    iamRoleStatements: N/A

  setStatusFailed:
    handler: src/handlers/setStatusFailed.default
    iamRoleStatements: N/A

  generateSubItemDoc:
    handler: src/handlers/generateSubItemDoc.default
    timeout: 300
    iamRoleStatements: N/A

  generateSubPackageDoc:
    handler: src/handlers/generateSubPackageDoc.default
    timeout: 300
    iamRoleStatements: N/A

  createPdf:
    image:
      name: docx_to_pdf_image
      command:
        - .build/src/handlers/createPdf.default
    environment:
      HOME: '/tmp'
    memorySize: 4096
    timeout: 300
    iamRoleStatements: N/A
        
  createPackagePdf:
    image:
      name: docx_to_pdf_image
      command:
        - .build/src/handlers/createPackagePdf.default
    environment:
      HOME: '/tmp'
    memorySize: 4096
    timeout: 600
    iamRoleStatements: N/A

  startPdfStateMachine:
    handler: src/handlers/startPdfStateMachine.default
    environment:
      STEP_FUNCTION_ARN: !Sub arn:aws:states:${self:provider.region}:${AWS::AccountId}:stateMachine:${self:service}-${sls:stage}-createPdf
    events:
      - eventBridge:
          eventBus: ${self:custom.eventBus.arn}
          pattern:
            source:
              - pdf.job
            detail-type:
              - create
            detail:
              status: [CREATED]
    iamRoleStatements: N/A

resources:
  - ${file(infrastructure-resources.yml)}

stepFunctions:
  stateMachines:
    createPdf:
      name: ${self:service}-${sls:stage}-createPdf
      alarms:
        topics:
          alarm: ${self:custom.alertTopicNameARN}
        metrics:
          - executionsTimedOut
          - executionsFailed
          - executionsAborted
          - metric: executionThrottled
      useExactVersion: true
      definition:
        StartAt: SetStateInProgress
        States:
          SetStateInProgress:
            Type: Task
            Resource: !GetAtt setStatusInProgress.Arn
            Next: GenerateDocChoice
          GenerateDocChoice:
            Type: Choice
            Choices:
              - Variable: $.type
                StringEquals: SUBMITTAL_ITEM
                Next: GenerateSubItemDoc
              - Variable: $.type
                StringEquals: SUBMITTAL_PACKAGE
                Next: GenerateSubPackageDoc
            Default: Fail
          GenerateSubPackageDoc:
            Type: Task
            Resource: !GetAtt generateSubPackageDoc.Arn
            Next: CreatePackagePdf
          GenerateSubItemDoc:
            Type: Task
            Resource: !GetAtt generateSubItemDoc.Arn
            Next: CreatePdf
          CreatePdf:
            Type: Task
            Resource: !GetAtt createPdf.Arn
            Next: SetStatusSucceeded
          CreatePackagePdf:
            Type: Task
            Resource: !GetAtt createPackagePdf.Arn
            Next: SetStatusSucceeded
          SetStatusSucceeded:
            Type: Task
            Resource: !GetAtt setStatusSucceeded.Arn
            End: true
          SetStatusFailed:
            Type: Task
            Resource: !GetAtt setStatusFailed.Arn
            Next: Fail
          Fail:
            Type: Fail

Command name and used flags

npx sls deploy --stage ${STAGE} --region 'us-west-2' --verbose

Command output

Deploying platform-async-pdf-svc to stage qa (us-west-2)

Using deployment bucket N/A
Compiling to node16 bundle with esbuild...
Compiling with concurrency: Infinity
Compiling completed.
Zip function: setStatusSucceeded - 357.59 KB [228 ms]
Zip function: createPdfJob - 363.17 KB [236 ms]
Zip function: publishDbEvents - 357.81 KB [237 ms]
Zip function: startPdfStateMachine - 357.65 KB [238 ms]
Zip function: setStatusInProgress - 357.59 KB [244 ms]
Zip function: setStatusFailed - 357.58 KB [247 ms]
Zip function: getPdfJob - 360.42 KB [249 ms]
Zip function: generateSubPackageDoc - 447.71 KB [249 ms]
Zip function: generateSubItemDoc - 447.71 KB [251 ms]

Error:
Encountered error during executing: docker build -t serverless-platform-async-pdf-svc-qa:docx_to_pdf_image -f /home/runner/work/platform-async-pdf-svc/platform-async-pdf-svc/Dockerfile ./
× Stack platform-async-pdf-svc-qa failed to deploy (13s)
Environment: linux, node 16.17.0, framework 3.22.0 (local), plugin 6.2.2, SDK 4.3.2
Credentials: Local, environment variables
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Output of the command:
Sending build context to Docker daemon  376.6MB

Step 1/10 : FROM public.ecr.aws/shelf/lambda-libreoffice-base:7.4-node16-x86_64 as builder
toomanyrequests: Rate exceeded

Error: Process completed with exit code 1.

Environment information

Framework Core: 3.22.0 (local)
Plugin: 6.2.2
SDK: 4.3.2

Start with the Dockerfile, serverless.yml, and the npx sls deploy command shown in the report. Reproduce the failure at the Docker build step and inspect the container-image deployment path for how the ECR rate-limit error is handled. Done means establishing whether the framework causes the failure and verifying the behavior against the reported command output.