bcgov / bcgov/wps

Dev: Rekey S3 objects from ISO date file names to ISO date paths

Open
#4,422 0 comments 0 reactions 0 assignees View on GitHub
4Refinement Task
Dominant language
Python
Stars
65
Forks
11
Avg merge
21h 25m
Merged PRs (30d)
70

Description

**Describe the task**
We have long lists of flat folders/files following the yyyy-mm-dd ISO naming scheme, however these are hard on the object store to list all at once, as well as visually inspect. We could instead store files based on a yyyy/mm key path scheme.

**Acceptance Criteria**
- [ ] Agree on a key path scheme for dates
- [ ] All existing data is migrated to the new scheme above
- [ ] All new data is stored using the new scheme above

**Additional context**
- Add any other context about the task here.
- Or here

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. First locate the S3 object-writing code and any existing data migration tooling, then confirm the proposed date key scheme with maintainers. Done means existing objects are migrated and new objects consistently use the agreed yyyy/mm path scheme.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.