aws-amplify / aws-amplify/amplify-cli

amplify push deployed an older version of package in lambda layer

Open
#12,916 8 comments 2 reactions 0 assignees View on GitHub
bug lambda-layers p2
Dominant language
TypeScript
Stars
2.9k
Forks
825
Avg merge
11d 23h
Merged PRs (30d)
2

Description

### How did you install the Amplify CLI?

npm

### If applicable, what version of Node.js are you using?

18

### Amplify CLI Version

12.0.3

### What operating system are you using?

Mac

### Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

no

### Describe the bug

We typically deploy to our prod backend environment through Amplify Console CD.

However, in this scenario I did an amplify push to our prod env on my local machine. I don't do this often, but gitlab was down so I did deployment from my machine. My machine has the latest code from git - the same code Amplify Console would have pulled.

What happened is a new lambda layer was made, however the lambda layer had an older version of a package we reference in the layer's package.json (e.g. v6.10.3). This is despite the actual local package.json (and the one in git referring to a newer version (v7).

This obviously caused an outage in our system because we have functionality that relies on the newer package version (v7) that is not in the old one.

What I think happened is the lambda layer created used the last package version I used when I last deployed to prod in my local machine. Again, I don't push to prod often via my local machine, so the last package pushed via my local machine was old. But I don't understand why it would do that when at the time of this latest push, my layer's package.json did refer to the new version of the package to be included.

Am I supposed to do an amplify pull before amplify push when pushing to an env I have not pushed to in a while (but other people / amplify console pushes to frequently?).

The interesting thing is the layer did not appear as needing any change yet amplify did create a new layer (or made our lambdas reference an older layer version, I'm not 100% sure):

![CleanShot 2023-07-07 at 12 49 40@2x](https://github.com/aws-amplify/amplify-cli/assets/81853/10d5eab0-82b2-4795-a5b5-4ebc2fe62989)

Another thing is we recently updated our Lambdas to Node 18, but perhaps the prod amplify push referenced an old lambda cloudformations with Node 16, so perhaps it used our last layer that was compatible with Node 16? Not sure.

### Expected behavior

amplify push should use latest code on machine to deploy and not use code from last deployment.

### Reproduction steps

I believe the repro steps would be:

1. amplify push via your local machine
2. Make a change (e.g. update the version of a package inside lambda layer's package.json)
3. amplify push via your machine
4. Make a change (e.g. update the version of a package inside lambda layer's package.json)
5.. amplify push via Amplify Console
3. Make a change (e.g. update graphql schema)
6. amplify push via your local machine

In step 6, the push will create a layer referencing the package.json in step 2, not step 4

### Project Identifier

9aa765f6154c6bafb93f14988a5c3497

### Log output

full log of the problematic deployment

```
# Put your logs below this line
➜ ess-app git:(master-redact) ✗ amplify push -y
⠙ Building resource api/EssGraphQLAPIBe careful when using @auth directives on a field in a root type...
⚠️
NodeToNodeEncryption is enabled for this Search Domain, disabling this flag or reverting to Amplify CLI <= 10.5.2 will result in this being disabled, triggering a rebuild of the Search Index. To backfill your search domain see https://docs.amplify.aws/cli/graphql/troubleshooting/#backfill-opensearch-index-from-dynamodb-table.

⚠️ WARNING: owners may reassign ownership for the following model(s) and role(s): . To read more: https://docs.amplify.aws/cli/graphql/authorization-rules/#per-user--owner-based-data-access.
⠸ Building resource api/EssGraphQLAPI✅ GraphQL schema compiled successfully.

Edit your schema at //amplify/backend/api/EssGraphQLAPI/schema.graphql or place .graphql files in a directory at //amplify/backend/api/EssGraphQLAPI/schema
✔ Successfully pulled backend environment prod from the cloud.
⠙ Building resource api/EssGraphQLAPIBe careful when using @auth directives on a field in a root type...
⚠️
NodeToNodeEncryption is enabled for this Search Domain, disabling this flag or reverting to Amplify CLI <= 10.5.2 will result in this being disabled, triggering a rebuild of the Search Index. To backfill your search domain see https://docs.amplify.aws/cli/graphql/troubleshooting/#backfill-opensearch-index-from-dynamodb-table.

⚠️ WARNING: owners may reassign ownership for the following model(s) and role(s): . If this is not intentional, you may want to apply field-level authorization rules to these fields. To read more: https://docs.amplify.aws/cli/graphql/authorization-rules/#per-user--owner-based-data-access.
⠸ Building resource api/EssGraphQLAPI✅ GraphQL schema compiled successfully.

Edit your schema at //amplify/backend/api/EssGraphQLAPI/schema.graphql or place .graphql files in a directory at //amplify/backend/api/EssGraphQLAPI/schema

Current Environment: prod

┌───────────┬───────────────────────────────────────────┬───────────┬───────────────────┐
│ Category │ Resource name │ Operation │ Provider plugin │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Api │ EssGraphQLAPI │ Update │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Auth │ essapp0f5644a0 │ Update │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ EssRestAPIFn │ Update │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Storage │ EssUsersS3 │ Update │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Analytics │ EssPinpoint │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Api │ EssRestAPI │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ EssDailyPatientRecallFn │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ EssDynamoStreamFn │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ EssGraphQLResolverFn │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ EssPtGraphQLResolverFn │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ EssS3TriggerFn │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ essapp0f5644a0CreateAuthChallenge │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ essapp0f5644a0DefineAuthChallenge │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ essapp0f5644a0PostConfirmation │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ essapp0f5644a0PreTokenGeneration │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ essapp0f5644a0VerifyAuthChallengeResponse │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function │ essappCliLambdaLayer │ No Change │ awscloudformation │
└───────────┴───────────────────────────────────────────┴───────────┴───────────────────┘
Be careful when using @auth directives on a field in a root type. @auth directives on field definitions use the source object to perform authorization logic and the source will be an empty object for fields on root types. Static group authorization should perform as expected....
NodeToNodeEncryption is enabled for this Search Domain, disabling this flag or reverting to Amplify CLI <= 10.5.2 will result in this being disabled, triggering a rebuild of the Search Index. To backfill your search domain see https://docs.amplify.aws/cli/graphql/troubleshooting/#backfill-opensearch-index-from-dynamodb-table.

⚠️ WARNING: owners may reassign ownership for the following model(s) and role(s): . If this is not intentional, you may want to apply field-level authorization rules to these fields. To read more: https://docs.amplify.aws/cli/graphql/authorization-rules/#per-user--owner-based-data-access.
✅ GraphQL schema compiled successfully.

Edit your schema at //amplify/backend/api/EssGraphQLAPI/schema.graphql or place .graphql files in a directory at //amplify/backend/api/EssGraphQLAPI/schema
⠸ Building resource api/EssGraphQLAPIBe careful when using @auth directives on a field in a root type. @auth directives on field definitions use the source object to perform authorization logic and the source will be an empty object for fields on root types. Static group authorization should perform as expected....
⚠️
NodeToNodeEncryption is enabled for this Search Domain, disabling this flag or reverting to Amplify CLI <= 10.5.2 will result in this being disabled, triggering a rebuild of the Search Index. To backfill your search domain see https://docs.amplify.aws/cli/graphql/troubleshooting/#backfill-opensearch-index-from-dynamodb-table.

⚠️ WARNING: owners may reassign ownership for the following model(s) and role(s): . If this is not intentional, you may want to apply field-level authorization rules to these fields. To read more: https://docs.amplify.aws/cli/graphql/authorization-rules/#per-user--owner-based-data-access.
⠴ Building resource api/EssGraphQLAPI✅ GraphQL schema compiled successfully.

Edit your schema at //amplify/backend/api/EssGraphQLAPI/schema.graphql or place .graphql files in a directory at //amplify/backend/api/EssGraphQLAPI/schema

Deployment completed.
Deploying root stack ess [ ======================================-- ] 17/18
amplify-ess-app-prod-150238 AWS::CloudFormation::Stack UPDATE_COMPLETE_CLEANUP_IN_PR… Fri Jul 07 2023 12:06:34…
functionessapp0f5644a0DefineA… AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:00:05…
functionessapp0f5644a0CreateA… AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:00:05…
functionessappCliLambdaLayer AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:00:40…
analyticsEssPinpoint AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:00:06…
functionessapp0f5644a0VerifyA… AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:01:17…
functionessapp0f5644a0PostCon… AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:01:17…
functionessapp0f5644a0PreToke… AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:01:18…
authessapp0f5644a0 AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:01:21…
apiEssGraphQLAPI AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:05:26…
functionEssS3TriggerFn AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:05:51…
functionEssDailyPatientRecall… AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:06:03…
functionEssPtGraphQLResolverFn AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:06:01…
storageEssUsersS3 AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:05:54…
functionEssGraphQLResolverFn AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:06:18…
functionEssDynamoStreamFn AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:06:31…
functionEssRestAPIFn AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:06:19…
apiEssRestAPI AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:06:22…
Deployed api EssGraphQLAPI [ ======================================== ] 18/18
AuthRolePolicy02A05DD8DC AWS::IAM::ManagedPolicy UPDATE_COMPLETE Fri Jul 07 2023 12:01:48…
GraphQLAPITransformerSchema3C… AWS::AppSync::GraphQLSchema UPDATE_COMPLETE Fri Jul 07 2023 12:01:47…
Referral AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:02:40…
FunctionDirectiveStack AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:02:27…
RxPaymentIntent AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:03:17…
MedicalHistory AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:03:18…
FormSubmission AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:03:18…
Rx AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:04:10…
RefundIntent AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:03:56…
PaymentIntent AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:04:10…
MedicationFollowup AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:03:45…
RxMedicationFollowup AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:03:57…
SearchableStack AWS::CloudFormation::Stack UPDATE_IN_PROGRESS Fri Jul 07 2023 12:04:14…
ConnectionStack AWS::CloudFormation::Stack UPDATE_COMPLETE Fri Jul 07 2023 12:04:36…
Deployed api EssRestAPI [ ======================================== ] 5/5
Deployed auth essapp0f5644a0 [ ======================================== ] 16/16
Deployed function EssDailyPatientRecallFn [ ======================================== ] 6/6
Deployed function EssDynamoStreamFn [ ======================================== ] 14/14
Deployed function EssGraphQLResolverFn [ ======================================== ] 4/4
Deployed function EssPtGraphQLResolverFn [ ======================================== ] 4/4
Deployed function EssRestAPIFn [ ======================================== ] 4/4
Deployed function EssS3TriggerFn [ ======================================== ] 4/4
Deployed function essapp0f5644a0CreateAuthChallenge [ ======================================== ] 3/3
Deployed function essapp0f5644a0DefineAuthChallenge [ ======================================== ] 3/3
Deployed function essapp0f5644a0PostConfirmation [ ======================================== ] 3/3
Deployed function essapp0f5644a0PreTokenGeneration [ ======================================== ] 4/4
Deployed function essapp0f5644a0VerifyAuthChallengeResponse [ ======================================== ] 3/3
LambdaFunction AWS::Lambda::Function UPDATE_IN_PROGRESS Fri Jul 07 2023 12:05:33…
Deployed function essappCliLambdaLayer [ ======================================== ] 3/3
LambdaLayerVersiona79f5a05 AWS::Lambda::LayerVersion UPDATE_COMPLETE Fri Jul 07 2023 12:00:33…
LambdaLayerPermissionAwsAccou… AWS::Lambda::LayerVersionPerm… UPDATE_COMPLETE Fri Jul 07 2023 12:00:37…
LambdaLayerPermissionPrivatea… AWS::Lambda::LayerVersionPerm… UPDATE_COMPLETE Fri Jul 07 2023 12:00:38…
Deployed storage EssUsersS3 [ ======================================== ] 11/11

✔ Generated GraphQL operations successfully and saved at frontend/projects/shared/graphql
✔ Code generated successfully and saved in file frontend/projects/shared/graphql/API.service.ts
Deployment state saved successfully.
Be careful when using @auth directives on a field in a root type. @auth directives on field definitions use the source object to perform authorization logic and the source will be an empty object for fields on root types. Static group authorization should perform as expected....

```

### Additional information

_No response_

### Before submitting, please confirm:

- [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
- [X] I have removed any sensitive information from my code snippets and submission.

Contributor guide

Open the contributing guide

Research direction

Start with the reported amplify push sequence and the deployment log, focusing on functionessappCliLambdaLayer and its CloudFormation stack entry. Compare the layer package.json and Node.js versions across the local, Amplify Console, and deployed environments. Done means a push consistently deploys the current package version rather than an older layer artifact.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, node.js, typescript
Domain
backend, cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.