aws-amplify / aws-amplify/amplify-cli
Updating amplify meta major performance issue
- Dominant language
- TypeScript
- Stars
- 2.9k
- Forks
- 825
- Avg merge
- 11d 23h
- Merged PRs (30d)
- 2
Description
### How did you install the Amplify CLI?
npm
### If applicable, what version of Node.js are you using?
18.7.1
### Amplify CLI Version
12.12.1
### What operating system are you using?
Mac
### Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
No manual changes made
### Describe the bug
The "store current cloud backend" step is extremely slow for larger code bases due to some inefficient code in the CLI. For larger projects this step can take several minutes to complete, in our case this step can take more than 15 minutes to complete.
The cause of this issue is found in the `update-amplify-meta.ts` file, in the `moveBackendResourcesToCurrentCloudBackend` function. During this step, ALL files from amplify resource folders are copied into the current cloud backend target directory (including node_modules, which in the case of functions with a lot of dependencies can potentially be a very large folder containing many files). After the copying the files over, a check is made to see if any node_modules were copied over (using glob.sync) after which they are deleted.
This is incredibly inefficient. Especially for larger projects this will drastically slow down deploy times.
Problematic code:
```TypeScript
const moveBackendResourcesToCurrentCloudBackend = (resources: $TSObject[]): void => {
const amplifyMetaFilePath = pathManager.getAmplifyMetaFilePath();
const amplifyCloudMetaFilePath = pathManager.getCurrentAmplifyMetaFilePath();
const backendConfigFilePath = pathManager.getBackendConfigFilePath();
const backendConfigCloudFilePath = pathManager.getCurrentBackendConfigFilePath();
const overridePackageJsonBackendFilePath = path.join(pathManager.getBackendDirPath(), 'package.json');
const overrideTsConfigJsonBackendFilePath = path.join(pathManager.getBackendDirPath(), 'tsconfig.json');
const overridePackageJsonCurrentCloudBackendFilePath = path.join(pathManager.getCurrentCloudBackendDirPath(), 'package.json');
const overrideTsConfigJsonCurrentCloudBackendFilePath = path.join(pathManager.getCurrentCloudBackendDirPath(), 'tsconfig.json');
for (const resource of resources) {
const sourceDir = path.normalize(path.join(pathManager.getBackendDirPath(), resource.category, resource.resourceName));
const targetDir = path.normalize(path.join(pathManager.getCurrentCloudBackendDirPath(), resource.category, resource.resourceName));
if (fs.pathExistsSync(targetDir)) {
fs.removeSync(targetDir);
}
fs.ensureDirSync(targetDir);
// in the case that the resource is being deleted, the sourceDir won't exist
if (fs.pathExistsSync(sourceDir)) {
fs.copySync(sourceDir, targetDir);
if (resource?.service === ServiceName.LambdaFunction || (resource?.service && resource?.service.includes('custom'))) {
removeNodeModulesDir(targetDir);
}
}
}
fs.copySync(amplifyMetaFilePath, amplifyCloudMetaFilePath, { overwrite: true });
fs.copySync(backendConfigFilePath, backendConfigCloudFilePath, { overwrite: true });
/**
* copying package.json and tsconfig.json to current cloud backend
*/
try {
fs.writeFileSync(overridePackageJsonCurrentCloudBackendFilePath, fs.readFileSync(overridePackageJsonBackendFilePath));
} catch (err) {
if (err.code !== 'ENOENT') {
throw err;
}
}
try {
fs.writeFileSync(overrideTsConfigJsonCurrentCloudBackendFilePath, fs.readFileSync(overrideTsConfigJsonBackendFilePath));
} catch (err) {
if (err.code !== 'ENOENT') {
throw err;
}
}
};
```
### Expected behavior
Storing current cloud backend shouldn't take more than 15 minutes.
### Reproduction steps
1. Have a repository with a large amount of resources with a node_modules folder
2. Run amplify push where changes have been made to all resources with node_modules (for maximum slowness)
### Project Identifier
_No response_
### Log output
```
# Put your logs below this line
```
### Additional information
_No response_
### Before submitting, please confirm:
- [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
- [X] I have removed any sensitive information from my code snippets and submission.
Contributor guide
Research direction
Start in update-amplify-meta.ts at moveBackendResourcesToCurrentCloudBackend and reproduce amplify push with large resource node_modules directories. Confirm that the current copy step processes those directories, then verify the revised behavior stores the cloud backend without copying unnecessary node_modules files and completes substantially faster.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- cli, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100