dandi / dandi/s3invsync

Beginning script at specific point in manifest.json

Open
#184 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3
Forks
1
PR merge metrics
No merged PRs in 30d

Description

@jwodder just filing an Issue to brainstorm:

When running the `s3invsync`, it takes quite some time to resume much further in the `manifest.json` where I would like a subsequent backup to continue -- I have been using the `--path-filter` as a human-readable mechanism, but I'm curious if you have any thoughts on how I can get `s3invsync` to start at a specific row/delimiter

I understand that it must at least do a regex comparison per row; however, wondering if there could be some sort of `f.seek()` behavior where a pointer could be set to start somewhere specific.

Curious to get your thoughts here -- for context, here is a sample command for what I am running, where the `PREFIX` is determined from an array of processes on MIT Engaging Cluster

```
s3invsync --path-filter "zarr/${PREFIX}[a-z0-9-]*" --allow-new-nonempty --ignore-errors all \
s3://linc-brain-mit-prod-us-east-2/linc-brain-mit-prod-us-east-2/production-configuration/ \
/orcd/data/linc/001/s3lincbrain/
```

Cc @kabilar @yarikoptic @satra

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how s3invsync reads manifest.json and applies --path-filter. The issue does not identify files, tests, or a settled behavior; first clarify the desired starting-point semantics and whether resuming at a row or delimiter is supported. Done would require an agreed design and corresponding implementation and tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cli, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.