duckdb / duckdb/duckdb-httpfs

Feature request / discussion: consolidate cache in DuckDB core

Open
#362 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

Hi team, nowadays we have cache implementation scattered in both [httpfs extension](https://github.com/duckdb/duckdb-httpfs/blob/2a6481d0bfc2cbf1a1f15d8076fe09b9d326d8d4/src/include/http_state.hpp#L25-L30) and [core](https://github.com/duckdb/duckdb/tree/main/src/include/duckdb/storage/external_file_cache), which
- Could lead to double caching
- Maintenance overhead, for example, there's deadlock bug in httpfs impl: https://github.com/duckdb/duckdb-httpfs/pull/335

I'm wondering if it's possible to make httpfs stateless as possible, which only responsible for IO communication with storage backend, and move all high-level features (like data caching) into core? Thank you!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the cache implementation in http_state.hpp with DuckDB core's src/include/duckdb/storage/external_file_cache, then read the linked httpfs pull request about the deadlock. The issue would be done only when the project agrees on and implements a clear consolidation design, including the intended httpfs/core responsibilities.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.