aspect-build / aspect-build/rules_aws

[FR]: Allow for directories and to "honor" structure

Open
#87 2 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Starlark
Stars
18
Forks
9
PR merge metrics
No merged PRs in 30d

Description

### What is the current behavior?

My team was working on a similar rule to allow uploading to S3. The rule here works for what it describes (it uses `cp` where we use `sync`). When run with `bazel run ...` the rule takes the given label and starts to `cp` each file it fines per the list.

### Describe the feature

The feature request is to keep the original underlying structure. Our team's use case is that we have directories with specific structures that we build out via Bazel to make a consistent build. As an example here's a setup for a Techdocs documentation site and the underlying tree that is generated by the tool with Bazel:

```shell
 bazel build //ns/vault:vault_docs
INFO: Invocation ID: 29b8ac8b-f6a9-432d-9209-00be609e3076
INFO: Analyzed target //ns/vault:vault_docs (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //ns/vault:vault_docs up-to-date:
bazel-bin/ns/vault/vault_docs_site
INFO: Elapsed time: 0.717s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

 tree bazel-bin/ns/vault/vault_docs_site
bazel-bin/ns/vault/vault_docs_site
├── 404.html
├── about
│   └── index.html
├── application_setup
│   └── index.html
├── assets
│   ├── images
│   │   └── favicon.png
│   ├── javascripts
│   │   ├── bundle.ad660dcc.min.js
│   │   ├── bundle.ad660dcc.min.js.map
│   │   ├── lunr
│   │   │   ├── min
│   │   │   │   ├── lunr.ar.min.js
│   │   │   │   ├── lunr.da.min.js
│   │   │   │   ├── lunr.de.min.js
│   │   │   │   ├── lunr.du.min.js
│   │   │   │   ├── lunr.el.min.js
│   │   │   │   ├── lunr.es.min.js
│   │   │   │   ├── lunr.fi.min.js
│   │   │   │   ├── lunr.fr.min.js
│   │   │   │   ├── lunr.he.min.js
│   │   │   │   ├── lunr.hi.min.js
│   │   │   │   ├── lunr.hu.min.js
│   │   │   │   ├── lunr.hy.min.js
│   │   │   │   ├── lunr.it.min.js
│   │   │   │   ├── lunr.ja.min.js
│   │   │   │   ├── lunr.jp.min.js
│   │   │   │   ├── lunr.kn.min.js
│   │   │   │   ├── lunr.ko.min.js
│   │   │   │   ├── lunr.multi.min.js
│   │   │   │   ├── lunr.nl.min.js
│   │   │   │   ├── lunr.no.min.js
│   │   │   │   ├── lunr.pt.min.js
│   │   │   │   ├── lunr.ro.min.js
│   │   │   │   ├── lunr.ru.min.js
│   │   │   │   ├── lunr.sa.min.js
│   │   │   │   ├── lunr.stemmer.support.min.js
│   │   │   │   ├── lunr.sv.min.js
│   │   │   │   ├── lunr.ta.min.js
│   │   │   │   ├── lunr.te.min.js
│   │   │   │   ├── lunr.th.min.js
│   │   │   │   ├── lunr.tr.min.js
│   │   │   │   ├── lunr.vi.min.js
│   │   │   │   └── lunr.zh.min.js
│   │   │   ├── tinyseg.js
│   │   │   └── wordcut.js
│   │   └── workers
│   │   ├── search.b8dbb3d2.min.js
│   │   └── search.b8dbb3d2.min.js.map
│   ├── stylesheets
│   │   ├── main.6543a935.min.css
│   │   ├── main.6543a935.min.css.map
│   │   ├── palette.06af60db.min.css
│   │   └── palette.06af60db.min.css.map
│   ├── vault-agent-overview.jpg
│   ├── vault-icon.png
│   └── vault-request.png
├── best_practices
│   └── index.html
├── index.html
├── kubernetes-setup
│   └── index.html
├── search
│   ├── lunr.js
│   ├── main.js
│   ├── search_index.json
│   └── worker.js
├── sitemap.xml
├── sitemap.xml.gz
├── techdocs_metadata.json
├── troubleshooting
│   └── index.html
├── vault-agent
│   └── index.html
└── vault_setup
└── index.html
```

The current target doesn't honor the tree structure, and instead flattens it and adds every file to the 'root' or the target `bucket`

```shell
 bazel run //ns/vault:vault_upload
INFO: Invocation ID: b25630d5-b978-4452-87f2-e77da5b72f49
INFO: Analyzed target //ns/vault:vault_upload (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //ns/vault:vault_upload up-to-date:
bazel-bin/ns/vault/vault_upload/s3_sync.sh
INFO: Elapsed time: 0.917s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/ns/vault/vault_upload/s3_sync.sh
Copying the following artifacts to s3://mybucket:
ns/vault/vault_docs_site

Copying ns/vault/vault_docs_site/404.html to s3://mybucket/404.html
upload: ns/vault/vault_docs_site/404.html to s3://mybucket/404.html
Copying ns/vault/vault_docs_site/about/index.html to s3://mybucketindex.html
upload: ns/vault/vault_docs_site/about/index.html to s3://mybucket/index.html
Copying ns/vault/vault_docs_site/application_setup/index.html to s3://mybucket/index.html
upload: ns/vault/vault_docs_site/application_setup/index.html to s3://mybucket/index.html
Copying ns/vault/vault_docs_site/assets/images/favicon.png to s3://mybucket/favicon.png
upload: ns/vault/vault_docs_site/assets/images/favicon.png to s3://mybucket/favicon.png
Copying ns/vault/vault_docs_site/assets/javascripts/bundle.ad660dcc.min.js to s3://mybucket/bundle.ad660dcc.min.js
upload: ns/vault/vault_docs_site/assets/javascripts/bundle.ad660dcc.min.js to s3://mybucket/bundle.ad660dcc.min.js
...
```

This limits the use of the rule to single files, or a bunch of files that are supposed to be under one directory at a given time. I think the rule becomes a lot more useful if it can copy the entire folder structure on top of the files itself, as s3 lets you do this (for example our team's own first try works with both single files and directories):

```shell
#!/usr/bin/env bash

set -eux

# Set the SOURCE location
readonly SOURCE="{{source}}"

# Set the Destination for SOURCE
readonly BUCKET_NAME="{{bucket_name}}"
readonly DESTINATION="{{destination}}"

# If the SOURCE is a directory we'll sync to avoid potential issues with copies
# A single file artifact we don't care as much so we'll stick to copying
if [ -d "$SOURCE" ]; then
aws s3 sync "${SOURCE}" "${BUCKET_NAME}/${DESTINATION}"
else
aws s3 cp "${SOURCE}" "${BUCKET_NAME}/${DESTINATION}"
fi

```

Contributor guide

Open the contributing guide

Research direction

Start by tracing the rule that generates the s3_sync.sh command shown in the issue and how it handles each artifact from the Bazel output tree. Implement directory-aware syncing while retaining relative paths, and verify the result against the demonstrated nested documentation tree and S3 destinations.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, bash
Domain
build-system, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.