aws / aws/jsii

Python: CdkFunction.environment is dereferenced prematurely instead of returning IResolvable

Open
#4,153 1 comment 0 reactions 0 assignees View on GitHub
bug p2
Dominant language
TypeScript
Stars
2.9k
Forks
267
Avg merge
1d 25m
Merged PRs (30d)
14

Description

### Describe the bug

[CdkFunction.environment](https://github.com/aws/aws-cdk/blob/784cd0e3303a/packages/aws-cdk-lib/aws-lambda/lib/function.ts#L807) is a lazily evaluated value which can be resolved to `CfnFunction.EnvironmentProperty`.

In TypeScript and Go, `environment` returns `LazyAny` and `typeregistry.anonymousProxy` respectively, both of which can then be resolved with `stack.resolve(environment)`.

In Python, `environment` returns an actual `EnvironmentProperty` instead of a reference, with `variables=None`. This resolves to an empty dict.

### Expected Behavior

I expected `CfnFunction.environment` to be an `IResolvable` value which could then be resolved with `stack.resolve(environment)`.

### Current Behavior

`CfnFunction.environment` is an empty `EnvironmentProperty` value with no reference, so it's not possible to get the environment variables from this value.

JSII_DEBUG output from Go:
```
> {"api":"get","property":"environment","objref":{"$jsii.byref":"aws-cdk-lib.aws_lambda.CfnFunction@10046"}}
[@jsii/kernel] get { '$jsii.byref': 'aws-cdk-lib.aws_lambda.CfnFunction@10046' } environment
[@jsii/kernel] value: LazyAny {
producer: { produce: [Function: produce] },
cache: false,
creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
options: {}
}
[@jsii/kernel] serialize LazyAny {
producer: { produce: [Function: produce] },
cache: false,
creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
options: {}
} {
serializationClass: 'Struct',
typeRef: {
type: { fqn: 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' },
optional: true
}
} {
serializationClass: 'RefType',
typeRef: { type: { fqn: 'aws-cdk-lib.IResolvable' }, optional: true }
}
[@jsii/kernel] Returning value type by reference
[@jsii/kernel] ret: {
'$jsii.byref': 'Object@10047',
'$jsii.interfaces': [ 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' ]
}
< {"ok":{"value":{"$jsii.byref":"Object@10047","$jsii.interfaces":["aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty"]}}}
```

From Python:
```
> {"objref":{"$jsii.byref":"aws-cdk-lib.aws_lambda.CfnFunction@10046"},"property":"environment","api":"get"}
[@jsii/kernel] get { '$jsii.byref': 'aws-cdk-lib.aws_lambda.CfnFunction@10046' } environment
[@jsii/kernel] value: LazyAny {
producer: { produce: [Function: produce] },
cache: false,
creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
options: {}
}
[@jsii/kernel] serialize LazyAny {
producer: { produce: [Function: produce] },
cache: false,
creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
options: {}
} {
serializationClass: 'Struct',
typeRef: {
type: { fqn: 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' },
optional: true
}
} {
serializationClass: 'RefType',
typeRef: { type: { fqn: 'aws-cdk-lib.IResolvable' }, optional: true }
}
[@jsii/kernel] Returning value type by reference
[@jsii/kernel] ret: {
'$jsii.byref': 'Object@10047',
'$jsii.interfaces': [ 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' ]
}
< {"ok":{"value":{"$jsii.byref":"Object@10047","$jsii.interfaces":["aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty"]}}}
> {"objref":{"$jsii.byref":"Object@10047"},"property":"variables","api":"get"}
[@jsii/kernel] get { '$jsii.byref': 'Object@10047' } variables
[@jsii/kernel] value: undefined
[@jsii/kernel] serialize undefined {
serializationClass: 'Map',
typeRef: { type: { collection: [Object] }, optional: true }
} {
serializationClass: 'RefType',
typeRef: { type: { fqn: 'aws-cdk-lib.IResolvable' }, optional: true }
}
[@jsii/kernel] ret: undefined
< {"ok":{}}
```

Note the extra request at the end of the Python logs. This is from the exact same line of code, `cfnFunction.Environment()` and `cfn_function.environment` respectively.

### Reproduction Steps

Paste each block into the stack file in the project created by `cdk init` for that language, then run `cdk ls`.

**Python:**
```py
from aws_cdk import Stack, aws_lambda as lambda_
from constructs import Construct

class CdkPyStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)

func = lambda_.Function(
self,
"Function",
code=lambda_.Code.from_inline("_"),
runtime=lambda_.Runtime.PYTHON_3_10,
handler="_",
environment={"KEY": "value"},
)

cfn_function = func.node.default_child
assert isinstance(cfn_function, lambda_.CfnFunction)
environment = cfn_function.environment
print(environment)
print(self.resolve(environment))
```
Output:
```
EnvironmentProperty()
{}
CdkPyStack
```

**TypeScript (working, for comparison):**
```ts
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';

export class CdkTsStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);

let func = new lambda.Function(this, "Function", {
code: lambda.Code.fromInline("_"),
runtime: lambda.Runtime.PYTHON_3_10,
handler: "_",
environment: { KEY: "value" },
});

let cfnFunction = func.node.defaultChild as lambda.CfnFunction;
let environment = cfnFunction.environment;
console.log(environment);
console.log(this.resolve(environment));
}
}
```
Output:
```
LazyAny {
producer: { produce: [Function: produce] },
cache: false,
creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
options: {}
}
{ variables: { KEY: 'value' } }
CdkTsStack
```

### Possible Solution

Here's what I've been able to figure out by stepping through with a debugger. Hopefully this helps.

`CfnFunction.environment` is a property which calls `jsii.get()`, aka `kernel.get()`.
https://github.com/aws/jsii/blob/14b5ed22fe87baf27be0f0ff61f6e423654baf39/packages/%40jsii/python-runtime/src/jsii/_kernel/__init__.py#L358-L366

The value returned from `kernel.get()` is an `ObjRef`, which is passed to ` _reference_map.resolve_reference()` via the `@_dereferenced` decorator and `_recursize_dereference()`.
https://github.com/aws/jsii/blob/14b5ed22fe87baf27be0f0ff61f6e423654baf39/packages/%40jsii/python-runtime/src/jsii/_kernel/__init__.py#L133-L151

The ref is something like `Object@10047`, so `class_fqn` is `Object`. The condition on line 104-106 below evaluates to True and it ends up creating and returning an `EnvironmentProperty` value with no data (because it hasn't been resolved with CDK yet).
https://github.com/aws/jsii/blob/14b5ed22fe87baf27be0f0ff61f6e423654baf39/packages/%40jsii/python-runtime/src/jsii/_reference_map.py#L102-L128

### Additional Information/Context

My use case is to create a [custom aspect](https://docs.aws.amazon.com/cdk/v2/guide/aspects.html) which adds default environment variables to all Lambdas in a stack. The aspect shouldn't overwrite variables set directly on a function, so it needs to know which ones already exist, if any.

### SDK version used

JSII 1.84.0, CDK lib and cli 2.84.0, Constructs 10.2.55

### Environment details (OS name and version, etc.)

Windows 10.0.14393 Build 14393, Python 3.11.3

Contributor guide

Open the contributing guide

Research direction

Reproduce the Python and TypeScript behavior in a CDK-initialized project with cdk ls, then inspect the jsii runtime entry points in packages/@jsii/python-runtime/src/jsii/_kernel/__init__.py and _reference_map.py. Compare how the ObjRef is dereferenced for CfnFunction.environment. Done means Python preserves the lazy IResolvable behavior so resolving the property exposes the configured environment variables rather than an empty EnvironmentProperty.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
developer-experience, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.