aws / aws/aws-cdk-cli

(cli): context overflow calculations are not correct

Open
#1,445 6 comments 0 reactions 0 assignees View on GitHub
bug p2
Dominant language
TypeScript
Stars
105
Forks
122
Avg merge
1d 17h
Merged PRs (30d)
71

Description

### Describe the bug

In https://github.com/aws/aws-cdk/pull/21373 we shipped functionality to limit the size of the context env variable by platform. Looking back at this now there seem to be a number of issues with the chosen approach:

- It appears that on Linux we are limited to 128 KB **per EnvVar** [1]. However the available space calculation looks at all variables, i.e. we are limiting too much.
- On Windows there seems to have been a historic limit **for all EnvVars** together [2]. However the available space calculation is only taking into account env vars set by the CDK CLI, i.e. we are not limiting enough.
- The Windows limitations seems to have been lifted on newer versions [2]. XP and Server 2003 being the last ones that had it. There is still a limit on 32 KB **per EnvVar** [3]. I.e. we are limiting too much unless we are on older windows versions.

---

### Available space calculation

https://github.com/aws/aws-cdk/blob/78fba2305c8e1710a3b83d810486a6875a82bf8b/packages/aws-cdk/lib/api/cxapp/exec.ts#L303-L311

This function is used to compute the space available for the context env var. It operates on bytes. The calculated value is then passed to a helper function to split a JS object by byte size into a main and overflow object.

The algorithm receives an input environment object. From this object, the byte sizes of all keys and values are summed up with the exception of the size of the value of the CDK Context. In other words, the computed size is roughly `size(input_env) - size(cdk_context_value)`. Finally the function subtracts the calculated "used up" space from an available limit and returns the value or 0 (if the used up space exceeds the limit).

Crucially, `input_env` is provided by the caller. So it isn't guaranteed to be a representation of the actual environment and could be a subset of variables. This would skew the result.

Also `limit` is a considered a total limit for __all env vars__. This makes the algorithm unusable for platforms that have a _per env var limit_.

---

[1] Linux env var limit: https://github.com/aws/aws-cdk/issues/19261#issue-1160668674
Environment strings in Linux are generally limited to 128K size and cannot be changed without recompiling the kernel!
MAX_ARG_STRLEN is a constant defined as PAGESIZE*32 in [include/uapi/linux/binfmts.h](https://github.com/torvalds/linux/blob/master/include/uapi/linux/binfmts.h). Its value cannot be changed without recompiling the kernel.

[2] Windows environment block limits: https://learn.microsoft.com/en-us/windows/win32/procthread/environment-variables
Windows Server 2003 and Windows XP: The maximum size of the environment block for the process is 32,767 characters. Starting with Windows Vista and Windows Server 2008, there is no technical limitation on the size of the environment block.

[3] Windows user user-defined environment variable limit: https://learn.microsoft.com/en-us/windows/win32/procthread/environment-variables
The maximum size of a user-defined environment variable is 32,767 characters.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.