kernelci / kernelci/kernelci-project

KernelCI mini-CDN (storage solution)

Open
#442 0 comments 0 reactions 0 assignees View on GitHub
techdebt
Dominant language
Python
Stars
14
Forks
32
Avg merge
1d 1h
Merged PRs (30d)
3

Description

Right now we are using Azure files storage, which is not very good for storing artifacts by multiple reasons:
* Very slow API to upload, list, delete many files
* We have no control over storage on downloads, we cannot check if someone have improper caching, or if someone is downloading too much
* Generated URL is "ugly" and not very user friendly
* Generally cloud storage is very expensive

We need to investigate better storage solution, that is likely will implement function of "mini-CDN".
* Keep "hot" artifacts in fast storage (block device)
* Keep "cold" artifacts that might be used from Azure in Azure files mounted by NFS to "CDN" VM
* Keep "cold" artifacts that might be used by other users on some cheap storages - Hetzner, etc.

This subject need to be investigated, and probably idea is improved.
Another prior research: https://github.com/kernelci/kernelci-project/issues/310

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the prior research in issue #310, then compare Azure Files with the proposed block-device and lower-cost storage options. Investigate how hot and cold artifacts could be served through a mini-CDN, including the NFS-mounted Azure Files path. Done means documenting an improved storage design and recommendation.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.