deepcelllineage / deepcelllineage/mitolin

Create an architecture for open source data exchange

Open
#9 0 comments 0 reactions 0 assignees View on GitHub
4-12 hours good first issue
Dominant language
Jupyter Notebook
Stars
2
Forks
4
PR merge metrics
No merged PRs in 30d

Description

## Aim

Design a straight forward, low barrier to entry, solution for the movement of data between open source collaborators of the DCL project.

## Background

We have a number of different systems that we are trying to make work together
- Individual's laptops
- [UCI hpc](https://hpc.oit.uci.edu/)
- Serge's GPU
- Google Drive

Data sizes vary. We want to be able to access and share data from 10s, 100s, and 1000s of files that can be as small as 20KB to as large as 10GB. Larger files can be processed on the HPC. We want a system to share small and medium sized files ASAP, which we can build on as our analyses get more data hungry.

### UCI HPC related docs

1. [How to transfer large amounts of data](http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html)
2. [OIT Transfering data](https://hpc.oit.uci.edu/transferring-data)
1. see examples
3. UCI Cloud File Storage Options
1. Google drive
1. [setup for UCI staff](https://www.oit.uci.edu/help/google/g-suite-setup/)
2. make folders public
4. [UCI Webfiles](https://www.oit.uci.edu/webfiles/)
1. not really what I want
2. How to set up Private and Public Directories in WebFiles v7: [UCI tutorial](https://uci.service-now.com/kb_view.do?sysparm_article=KB0010526)
3. upload files [UCI tutorial](https://uci.service-now.com/kb_view.do?sysparm_article=KB0010527)

## Questions?

Please put questions related to this issue in this issue thread.

If you want a quick response, post a link to your comment in this thread to Slack #deepcelllineage and tag @Deena or others. You can also DM @Deena.

To join Slack enter your email address [here](http://bit.ly/JoinSlackFastaiSFbay). For questions NOT specifically related to this issue, get in touch through any of the communication methods listed in [DCL's overview README](https://github.com/deepcelllineage/overview/blob/master/README.md).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the linked UCI HPC transfer, Google Drive, and WebFiles documentation, then compare those options against the stated file-size and collaboration requirements. Done means documenting a low-barrier data-exchange architecture that supports laptops, HPC, GPU, and shared storage, with a clear path from small files to larger datasets.

Written by the indexing model from the issue text.

Assessment

Domain
cloud, data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.