AlexsLemonade / AlexsLemonade/training-modules

Expand data maintenance documentation

Open
#973 2 comments 0 reactions 1 assignee Claimed by @allyhawkins View on GitHub
Dominant language
HTML
Stars
77
Forks
35
Avg merge
1d 13h
Merged PRs (30d)
4

Description

> Something I have _not_ yet done here is add the osteo data to the sync/link scripts, and also files aren't yet on S3, but I want to do this before merge (note also I forgot to stack this on #971 so there will be very smol conflicts). This is because I wanted to make sure to ask the `syncup-s3.sh` script which I realize I'm slightly confused by - why is this written to run from the training-modules directory instead of from the shared/data/training-modules directory? I feel I'm missing a logical connection here and looking for some clarification, thanks!

I don't think I am going to review this today, but I wanted to reply about what I remember here.

The idea, as I recall, is that you are probably doing your processing in your personal copy of `training-modules`. So you do all the setup work there, then sync up your changes to the s3 bucket. No `sudo` needed, importantly. Also important, because it uses `aws s3 sync`, you don't need to have all of the files; it will leave things on the server even if you didn't recreate them.

Then, when everything works as expected, we can use `aws s3 sync "s3://alsf-datalab-training-data/training-modules" /shared/data/training-modules` to get everything to the shared data directory.

That is my memory, at least. It's been a while since we had a new module/new data to sync!

_Originally posted by @jashapiro in https://github.com/AlexsLemonade/training-modules/issues/972#issuecomment-4270513985_

---

Josh has offered a very streamlined explanation for (his memory, at least) data prep processes, and this is certainly clearer than some of the docs we have. So, we should get some of this content into docs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.