serverless / serverless/serverless

Maximum policy size of 10240 bytes exceeded with lots of schedule events

Open
#10,357 11 comments 0 reactions 1 assignee View on GitHub

@simonarcher99 is already working on this.

Since May 13, 2022.

bug/design cat/aws-event-schedule good first issue help wanted
Dominant language
JavaScript
Stars
46.9k
Forks
5.7k
Avg merge
10h 7m
Merged PRs (30d)
57

Description

Are you certain it's a bug?
  • Yes, it looks like a bug
Is the issue caused by a plugin?
  • It is not a plugin issue
Are you using the latest version?
  • Yes, I'm using the latest version
Is there an existing issue for this?
  • I have searched existing issues, it hasn't been reported yet
Issue description

When adding multiple schedule events to one Lambda function, an individual permission statement is created for each schedule to allow it to invoke the Lambda function. This results in a large policy document that eventually exceeds AWS's 10KB limit. In my specific case, >=29 schedules were enough to cause the issue.

These individual permissions could be replaced with a single permission containing a list of resources. The same scenario has occurred in a couple of other areas in the past, and has been resolved with this method:

Service configuration (serverless.yml) content
functions:
  dataWarehouseExport:
    handler: dataWarehouseExport
    memorySize: 10240
    layers:
      - ${self:custom.runtimeLayer}
    events:
      - schedule:
          rate: cron(1 3 * * ? *) # Daily, at 03:01
          input:
            command: app:data-warehouse:REDACTED0
      - schedule:
          rate: cron(2 3 * * ? *) # Daily, at 03:02
          input:
            command: app:data-warehouse:REDACTED1
      - schedule:
          rate: cron(3 3 * * ? *) # Daily, at 03:03
          input:
            command: app:data-warehouse:REDACTED2
      - schedule:
          rate: cron(4 3 * * ? *) # Daily, at 03:04
          input:
            command: app:data-warehouse:REDACTED3
      - schedule:
          rate: cron(6 3 * * ? *) # Daily, at 03:06
          input:
            command: app:data-warehouse:REDACTED4
      - schedule:
          rate: cron(7 3 * * ? *) # Daily, at 03:07
          input:
            command: app:data-warehouse:REDACTED5
      - schedule:
          rate: cron(5 3 * * ? *) # Daily, at 03:05
          input:
            command: app:data-warehouse:REDACTED6
      - schedule:
          rate: cron(8 3 * * ? *) # Daily, at 03:08
          input:
            command: app:data-warehouse:REDACTED7
      - schedule:
          rate: cron(9 3 * * ? *) # Daily, at 03:09
          input:
            command: app:data-warehouse:REDACTED8
      - schedule:
          rate: cron(10 3 * * ? *) # Daily, at 03:10
          input:
            command: app:data-warehouse:REDACTED9
      - schedule:
          rate: cron(11 3 * * ? *) # Daily, at 03:11
          input:
            command: app:data-warehouse:REDACTED10
      - schedule:
          rate: cron(12 3 * * ? *) # Daily, at 03:12
          input:
            command: app:data-warehouse:REDACTED11
      - schedule:
          rate: cron(13 3 * * ? *) # Daily, at 03:13
          input:
            command: app:data-warehouse:REDACTED12
      - schedule:
          rate: cron(14 3 * * ? *) # Daily, at 03:14
          input:
            command: app:data-warehouse:REDACTED13
      - schedule:
          rate: cron(15 3 * * ? *) # Daily, at 03:15
          input:
            command: app:data-warehouse:REDACTED14
      - schedule:
          rate: cron(16 3 * * ? *) # Daily, at 03:16
          input:
            command: app:data-warehouse:REDACTED15
      - schedule:
          rate: cron(17 3 * * ? *) # Daily, at 03:17
          input:
            command: app:data-warehouse:REDACTED16
      - schedule:
          rate: cron(18 3 * * ? *) # Daily, at 03:18
          input:
            command: app:data-warehouse:REDACTED17
      - schedule:
          rate: cron(19 3 * * ? *) # Daily, at 03:19
          input:
            command: app:data-warehouse:REDACTED18
      - schedule:
          rate: cron(20 3 * * ? *) # Daily, at 03:20
          input:
            command: app:data-warehouse:REDACTED19
      - schedule:
          rate: cron(21 3 * * ? *) # Daily, at 03:21
          input:
            command: app:data-warehouse:REDACTED20
      - schedule:
          rate: cron(22 3 * * ? *) # Daily, at 03:22
          input:
            command: app:data-warehouse:REDACTED21
      - schedule:
          rate: cron(23 3 * * ? *) # Daily, at 03:23
          input:
            command: app:data-warehouse:REDACTED22
      - schedule:
          rate: cron(24 3 * * ? *) # Daily, at 03:24
          input:
            command: app:data-warehouse:REDACTED23
      - schedule:
          rate: cron(25 3 * * ? *) # Daily, at 03:25
          input:
            command: app:data-warehouse:REDACTED24
      - schedule:
          rate: cron(26 3 * * ? *) # Daily, at 03:26
          input:
            command: app:data-warehouse:REDACTED25
      - schedule:
          rate: cron(27 3 * * ? *) # Daily, at 03:27
          input:
            command: app:data-warehouse:REDACTED26
      - schedule:
          rate: cron(43 3 * * ? *) # Daily, at 03:43
          input:
            command: app:data-warehouse:REDACTED27
      - schedule:
          rate: cron(46 3 * * ? *) # Daily, at 03:46
          input:
            command: app:data-warehouse:REDACTED28
      - schedule:
          rate: cron(29 3 * * ? *) # Daily, at 03:29
          input:
            command: app:data-warehouse:REDACTED29
      - schedule:
          rate: cron(28 3 * * ? *) # Daily, at 03:28
          input:
            command: app:data-warehouse:REDACTED30
      - schedule:
          rate: cron(30 3 * * ? *) # Daily, at 03:30
          input:
            command: app:data-warehouse:REDACTED31
      - schedule:
          rate: cron(0 4 * * ? *) # Daily, at 04:00
          input:
            command: app:data-warehouse:REDACTED32
      - schedule:
          rate: cron(31 3 * * ? *) # Daily, at 03:31
          input:
            command: app:data-warehouse:REDACTED33
      - schedule:
          rate: cron(32 3 * * ? *) # Daily, at 03:32
          input:
            command: app:data-warehouse:REDACTED34
      - schedule:
          rate: cron(33 3 * * ? *) # Daily, at 03:33
          input:
            command: app:data-warehouse:REDACTED35
      - schedule:
          rate: cron(34 3 * * ? *) # Daily, at 03:34
          input:
            command: app:data-warehouse:REDACTED36
      - schedule:
          rate: cron(35 3 * * ? *) # Daily, at 03:35
          input:
            command: app:data-warehouse:REDACTED37
      - schedule:
          rate: cron(36 3 * * ? *) # Daily, at 03:36
          input:
            command: app:data-warehouse:REDACTED38
      - schedule:
          rate: cron(44 3 * * ? *) # Daily, at 03:44
          input:
            command: app:data-warehouse:REDACTED39
      - schedule:
          rate: cron(41 3 * * ? *) # Daily, at 03:41
          input:
            command: app:data-warehouse:REDACTED40
      - schedule:
          rate: cron(37 3 * * ? *) # Daily, at 03:37
          input:
            command: app:data-warehouse:REDACTED41
      - schedule:
          rate: cron(42 3 * * ? *) # Daily, at 03:42
          input:
            command: app:data-warehouse:REDACTED42
      - schedule:
          rate: cron(38 3 * * ? *) # Daily, at 03:38
          input:
            command: app:data-warehouse:REDACTED43
      - schedule:
          rate: cron(39 3 * * ? *) # Daily, at 03:39
          input:
            command: app:data-warehouse:REDACTED44
      - schedule:
          rate: cron(40 3 * * ? *) # Daily, at 03:40
          input:
            command: app:data-warehouse:REDACTED45
      - schedule:
          rate: cron(45 3 * * ? *) # Daily, at 03:45
          input:
            command: app:data-warehouse:REDACTED46
Command name and used flags

sls deploy -s sandbox

Command output
…REDACTED…

 Serverless Error ----------------------------------------
 
  An error occurred: DataWarehouseExportLambdaPermissionEventsRuleSchedule29 - The final policy size (20764) is bigger than the limit (20480). (Service: AWSLambda; Status Code: 400; Error Code: PolicyLengthExceededException; Request ID: a9945e0b-d7c2-4bbd-b34c-e323bd82a3ff; Proxy: null).
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              12.22.7
     Framework Version:         2.66.2
     Plugin Version:            5.5.1
     SDK Version:               4.3.0
     Components Version:        3.18.1
Environment information
Framework Core: 2.66.2
Plugin: 5.5.1
SDK: 4.3.0
Components: 3.18.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.