[Proposed Feature] Object-level Deduplication to Reduce Storage Costs and Redundant Uploads

Description of Feature:

Problem Description

As the platform grows, it is likely that different users (or organizations) will upload identical files, particularly when working with common open datasets.

Currently, if User A and User B upload the exact same file to their respective repositories, it appears that Source Cooperative stores two copies. This increases storage costs for the platform and bandwidth usage for the user.

Specific Use Case:
I am currently working with elevation data related to the mapterhorn project

• Mapterhorn currently hosts processed elevation data but not the source extracted tarballs.
• They have mentioned they might mirror these tarballs in the future (see discussion here.
• I intend to upload these extracted tarballs to my own organization now.
• My understanding is that if Mapterhorn decides to upload the extracted tarballs later, Source Cooperative effectively pays double the storage cost for the same data.

Proposed Solution

I propose implementing a deduplication check during the upload process to handle these collisions efficiently.

The Workflow:

1. Calculate Hash: Before uploading, the client (CLI/script) calculates a unique hash of the file (e.x. MD5, SHA-256, etc.).
2. Check Existence: The client sends this hash to the Source Cooperative API to see if that specific sequence of bytes already exists on the platform.
3. Conditional Upload:

• If a match is found: The system skips the physical upload. Instead, it creates a reference in my repository pointing to the existing data object.
• If no match is found: The client proceeds to upload the file normally.

Handling Deletion

This feature would require a "shared" ownership model for the underlying data:

• If User A (the original uploader) deletes the file from their repository, but User B (me) still has a reference to that same file hash, the actual data should not be deleted from the backend storage.
• The system should only remove the link from User A's view. The physical data should persist until no repositories reference that specific hash anymore.

What value is this feature adding to Source Cooperative?

• Reduced Costs: significantly lower storage bills for Source Cooperative by avoiding redundant copies of popular datasets.
• Faster Uploads: Users attempting to upload known public datasets will see "instant" completion if the data is already on the platform.

Start by reviewing the upload client, Source Cooperative API, and backend storage behavior described in the issue. Clarify the deduplication and shared-ownership design, including hash checks, references, and deletion; done would require an agreed implementation plan for these cross-cutting changes.