GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Copying esm2015 folders with rsync

Open
#1,047 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

I am trying to copy the esm2015 folders generated with Angular 9 into a cloud storage bucket. I'm using the following cloud build configuration:
```yml
steps:
# Create node_modules
- name: "ubuntu"
args: ["mkdir", "-p", "node_modules"]

# Copy node_modules from Cloud Storage
- name: gcr.io/cloud-builders/gsutil
args: ["-m", "cp", "-r", "gs://cache-web-ruydo-dev/web", "node_modules"]

# Install node packages
- name: node:10.17.0
entrypoint: npm
args: ["install"]

# Build productive files
- name: node:10.17.0
entrypoint: npm
args: ["run", "build", "--", "--configuration=dev"]

# Copy node_modules to Cloud Storage
- name: gcr.io/cloud-builders/gsutil
args:
[
"-m",
"rsync",
"-r",
"-x",
'"^((?!\/esm2015\/).)*$"',
"node_modules",
"gs://cache-web-ruydo-dev/web",
]

# Deploy to google cloud app engine
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy", "app.yaml", "--version=prod"]

timeout: "1200s" # compile timeout
```

I am not sure if the pattern is incorrectly written or not, I have tested it a lot of times in different regex applications.
https://www.regextester.com/15
https://regexr.com/

It does seem to match any file that does not contain "/esm2015/", however, when I try it in cloud build it instead copies every file regardless of the pattern.
```
Copying file://node_modules/@angular/cdk/drag-drop/directives/drag-placeholder.d.ts [Content-Type=video/MP2T]...
\ [369 files][ 11.7 MiB/195.6 MiB] 700.7 KiB/s ETA 00:04:29
Copying file://node_modules/@angular/cdk/drag-drop/directives/drag-preview.d.ts.__ivy_ngcc_bak [Content-Type=application/octet-stream]...
\ [370 files][ 11.7 MiB/195.6 MiB] 659.8 KiB/s ETA 00:04:45
Copying file://node_modules/@angular/cdk/drag-drop/directives/drag.d.ts [Content-Type=video/MP2T]...
| [372 files][ 11.7 MiB/195.6 MiB] 551.7 KiB/s ETA 00:05:41
| [374 files][ 11.7 MiB/195.6 MiB] 553.3 KiB/s ETA 00:05:40
Copying file://node_modules/@angular/cdk/drag-drop/directives/drop-list.d.ts [Content-Type=video/MP2T]...
| [376 files][ 11.7 MiB/195.6 MiB] 556.3 KiB/s ETA 00:05:38
| [377 files][ 11.7 MiB/195.6 MiB] 549.4 KiB/s ETA 00:05:43
| [378 files][ 11.8 MiB/195.6 MiB] 551.3 KiB/s ETA 00:05:42
| [379 files][ 11.8 MiB/195.6 MiB] 550.1 KiB/s ETA 00:05:42
Copying file://node_modules/@angular/cdk/drag-drop/drag-drop-registry.d.ts.__ivy_ngcc_bak [Content-Type=application/octet-stream]...
| [381 files][ 11.8 MiB/195.6 MiB] 551.1 KiB/s ETA 00:05:42
| [382 files][ 11.8 MiB/195.6 MiB] 551.0 KiB/s ETA 00:05:42
| [383 files][ 11.8 MiB/195.6 MiB] 543.6 KiB/s ETA 00:05:46
Copying file://node_modules/@angular/cdk/drag-drop/drag-parent.d.ts [Content-Type=video/MP2T]...
Copying file://node_modules/@angular/cdk/drag-drop/drag-ref.d.ts [Content-Type=video/MP2T]...
/ [386 files][ 11.8 MiB/195.6 MiB] 546.0 KiB/s ETA 00:05:45
/ [387 files][ 11.8 MiB/195.6 MiB] 499.2 KiB/s ETA 00:06:17
Copying file://node_modules/@angular/cdk/drag-drop/drag-utils.d.ts [Content-Type=video/MP2T]...
Copying file://node_modules/@angular/cdk/drag-drop/drop-list-ref.d.ts [Content-Type=video/MP2T]...
/ [388 files][ 11.8 MiB/195.6 MiB] 409.2 KiB/s ETA 00:07:40
/ [391 files][ 11.8 MiB/195.6 MiB] 411.4 KiB/s ETA 00:07:38
/ [392 files][ 11.8 MiB/195.6 MiB] 410.5 KiB/s ETA 00:07:39
Copying file://node_modules/@angular/cdk/drag-drop/parent-position-tracker.d.ts [Content-Type=video/MP2T]...
Copying file://node_modules/@angular/cdk/drag-drop/public-api.d.ts [Content-Type=video/MP2T]...
Copying file://node_modules/@angular/cdk/drag-drop/transition-duration.d.ts [Content-Type=video/MP2T]...
/ [393 files][ 11.8 MiB/195.6 MiB] 392.3 KiB/s ETA 00:08:00
/ [397 files][ 11.8 MiB/195.6 MiB] 388.8 KiB/s ETA 00:08:04
Copying file://node_modules/@angular/cdk/esm2015/a11y/aria-describer/aria-reference.js [Content-Type=application/javascript]...
- [398 files][ 11.8 MiB/195.6 MiB] 309.4 KiB/s ETA 00:10:08
- [401 files][ 11.8 MiB/195.6 MiB] 313.5 KiB/s ETA 00:10:00
Copying file://node_modules/@angular/cdk/esm2015/a11y/focus-monitor/focus-monitor.js [Content-Type=application/javascript]...
- [402 files][ 11.8 MiB/195.6 MiB] 312.4 KiB/s ETA 00:10:02
- [403 files][ 11.9 MiB/195.6 MiB] 289.4 KiB/s ETA 00:10:50
Copying file://node_modules/@angular/cdk/esm2015/a11y/focus-trap/configurable-focus-trap-factory.js [Content-Type=application/javascript]...
Copying file://node_modules/@angular/cdk/esm2015/a11y/focus-trap/event-listener-inert-strategy.js [Content-Type=application/javascript]...
- [406 files][ 11.9 MiB/195.6 MiB] 290.9 KiB/s ETA 00:10:47
- [407 files][ 12.0 MiB/195.6 MiB] 290.5 KiB/s ETA 00:10:48
\ [408 files][ 12.0 MiB/195.6 MiB] 226.1 KiB/s ETA 00:13:52
\ [411 files][ 12.0 MiB/195.6 MiB] 223.3 KiB/s ETA 00:14:02
Copying file://node_modules/@angular/cdk/esm2015/a11y/high-contrast-mode/high-contrast-mode-detector.js [Content-Type=application/javascript]...
\ [411 files][ 12.0 MiB/195.6 MiB] 222.8 KiB/s ETA 00:14:04
\ [412 files][ 12.0 MiB/195.6 MiB] 222.3 KiB/s ETA 00:14:06
\ [413 files][ 12.1 MiB/195.6 MiB] 232.5 KiB/s ETA 00:13:28
Copying file://node_modules/@angular/cdk/esm2015/a11y/key-manager/activedescendant-key-manager.js [Content-Type=application/javascript]...
Copying file://node_modules/@angular/cdk/esm2015/a11y/key-manager/list-key-manager.js [Content-Type=application/javascript]...
Copying file://node_modules/@angular/cdk/esm2015/a11y/live-announcer/live-announcer-tokens.js [Content-Type=application/javascript]...
\ [414 files][ 12.1 MiB/195.6 MiB] 204.6 KiB/s ETA 00:15:19
\ [418 files][ 12.1 MiB/195.6 MiB] 188.9 KiB/s ETA 00:16:35
Copying file://node_modules/@angular/cdk/esm2015/accordion/accordion-item.js [Content-Type=application/javascript]...
```

Contributor guide

Open the contributing guide

Research direction

Start with the gsutil rsync entry point and the documented -x exclusion pattern, using the Cloud Build configuration and node_modules example from the report. Reproduce the command with an esm2015 directory and verify whether those files are excluded; done means the sync copies other files while skipping paths containing /esm2015/.

Written by the indexing model from the issue text.

Assessment

Tech stack
angular, google-cloud
Domain
cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.