nextcloud / nextcloud/server

Store metadata in S3 alongside the blobs

Open
#30,824 4 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

0. Needs triage enhancement feature: filesystem feature: object storage
Dominant language
PHP
Stars
36.9k
Forks
5.2k
Avg merge
2d 3h
Merged PRs (30d)
713

Description

Is your feature request related to a problem? Please describe.
yes. when storing data in S3, there is not metadata. this makes life very difficult if filecache table and s3 state disagree (e.g. about last modified date, size or sha1).

Describe the solution you'd like
it would be great to have metadata in S3 alongside the blobs. this might be tags or separate objects. tags might be a bit more obvious, but separate objects should be more robust - a long path in ancient thai script might not be an ideal value for a tag.

so, either:

   urn:oid:223
   AWS Tags:
      - Path: files/foobar.xml
      - SHA1: da39a3ee5e6b4b0d3255bfef95601890afd80709
      - LastModified: 12234455   <<< because the last modified in S3 might be different

or:

   urn:oid:223
   urn:oid:223.json
   { 
          Path: ".....",
          Sha1: "da39a3ee5e6b4b0d3255bfef95601890afd80709",
          LastModified: 123345667
   } 

it would also be great if files:scan, files:cleanup and files:checksum would deal with these.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the S3 storage implementation and the entry points for files:scan, files:cleanup, and files:checksum. Compare the requested metadata options and define how metadata should remain consistent with the filecache table; done means the selected approach works for S3 blobs and all three commands handle it.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cli, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.