Avoiding metadata bloat caused by many long URLs
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I woke up this morning wondering whether it would be possible to allow a variable to be defined in the referencefile spec. I'm worried about long s3 (or other) urls bloating the metadata.
If we could do something like:
prefix001='first/superlongurl/that/keeps/going/on/and/on/for/ever/to/some/dir'
prefix002='second/superlongurl/that/keeps/going/on/and/on/for/ever/to/some/dir'
"key1": {
["s3://$prefix001/data001.nc", 10000, 100]
}
"key2": {
["s3://$prefix001/data001.nc", 10100, 100]
}
"key3": {
["s3://$prefix002/data001.nc", 10000, 100]
}
"key4": {
["s3://$prefix002/data001.nc", 10100, 100]
}
we could make the bloat a lot smaller
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the referencefile spec and tracing how its metadata entries represent repeated URLs. Define how variables should be declared and expanded in the shown reference format, then verify that repeated long prefixes are stored compactly without changing the resolved references.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100