apache / apache/libcloud

Support for Cloudflare's D1 and/or R2

Open
#1,865 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
931
Avg merge
1d 2h
Merged PRs (30d)
4

Description

## Feature Description

Cloudflare has been developing an alternative to traditional storage providers, with the benefit that it has no egress fees for the data.

- [**D1**](https://developers.cloudflare.com/d1) is a relational database build on SQLite, which currently is on Alpha. It can be access and manipulated through the online GUI on [Cloudflare's Dashboard](https://dash.cloudflare.com/) or through the CLI using a npm package called [wrangler](https://github.com/cloudflare/workers-sdk/tree/main/packages/wrangler).
- [**R2**](https://developers.cloudflare.com/r2) is the Cloudflare's implementation of an object storage service. It can be used through the GUI on [Cloudflare's Dashboard](https://dash.cloudflare.com/), through the CLI with [wrangler](https://github.com/cloudflare/workers-sdk/tree/main/packages/wrangler), or using the [Amazon's S3 API](https://aws.amazon.com/s3/).

## Proposal

1. First the easiest, R2. To begin with, test if Apache Libcloud [`libcloud.storage.drivers.s3`](https://libcloud.readthedocs.io/en/stable/apidocs/libcloud.storage.drivers.html#module-libcloud.storage.drivers.s3) works correctly with an R2 bucket, and if so, update the documentation to reflect that said module can be used with R2. It would be a good idea to document what functions are NOT available, according to Cloudflare's R2 [documentation](https://developers.cloudflare.com/r2/data-access/s3-api/api/).
2. If `libcloud.storage.drivers.s3` doesn't work with R2, another module can be created (`libcloud.storage.drivers.r2`, for example), which is based on the S3's module but with a few adjustments, and removing functions that are unavailable on R2. I do think this is a cleaner solution, to keep S3 separated from R2.
3. Finally, as Apache Libcloud does not (yet) manage Cloud Databases, a fine addition to the library would be to start a new module (maybe `libcloud.sql.*`?) and implement certain popular SQL Databases, such as [Amazon's RDS](https://aws.amazon.com/rds/), [Azure's SQL Database](https://azure.microsoft.com/en-us/products/azure-sql/database/), and of course Cloudflare's D1.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing libcloud.storage.drivers.s3 against Cloudflare R2's S3 API documentation and determine whether existing compatibility can be documented or a separate libcloud.storage.drivers.r2 module is needed. The broader proposal also suggests designing libcloud.sql.* for D1 and other managed SQL databases; done requires an agreed scope, implementation, tests, and documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, sql, sqlite
Domain
cloud, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.