Store metadata in S3 alongside the blobs
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 36.9k
- Forks
- 5.2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 713
Description
Is your feature request related to a problem? Please describe.
yes. when storing data in S3, there is not metadata. this makes life very difficult if filecache table and s3 state disagree (e.g. about last modified date, size or sha1).
Describe the solution you'd like
it would be great to have metadata in S3 alongside the blobs. this might be tags or separate objects. tags might be a bit more obvious, but separate objects should be more robust - a long path in ancient thai script might not be an ideal value for a tag.
so, either:
urn:oid:223
AWS Tags:
- Path: files/foobar.xml
- SHA1: da39a3ee5e6b4b0d3255bfef95601890afd80709
- LastModified: 12234455 <<< because the last modified in S3 might be different
or:
urn:oid:223
urn:oid:223.json
{
Path: ".....",
Sha1: "da39a3ee5e6b4b0d3255bfef95601890afd80709",
LastModified: 123345667
}
it would also be great if files:scan, files:cleanup and files:checksum would deal with these.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the S3 storage implementation and the entry points for files:scan, files:cleanup, and files:checksum. Compare the requested metadata options and define how metadata should remain consistent with the filecache table; done means the selected approach works for S3 blobs and all three commands handle it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cli, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100