duckdb / duckdb/duckdb-httpfs

When cache_httpfs is loaded, HTTP response headers (e.g., Content-Range) are not captured in HTTP logging (duckdb_logs_parsed('HTTP'))

Open
#345 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

**Description:**

When cache_httpfs is loaded, HTTP response headers (e.g., Content-Range) are not captured in HTTP logging (duckdb_logs_parsed('HTTP')).

**Root Cause:**

cache_httpfs registers a CacheFileSystem that wraps httpfs. On cache miss, it calls internal_filesystem->Read() directly on the httpfs file handle. This invokes HTTPFileSystem::ReadInternal → GetRangeRequest which performs the HTTP GET, but the streaming response path in httpfs_httplib_client.cpp does not propagate response headers to the final HTTPResponse object that LogRequest receives.

**Code path comparison:**

Without cache_httpfs (works):
HTTPFileSystem::OpenFile → HTTPUtil::RunRequestWithRetry → LogRequest(request, response) ✓

With cache_httpfs (broken):
CacheFileSystem::Read → DiskCacheReader::ReadAndCache → internal_filesystem->Read()
→ HTTPFileSystem::ReadInternal → GetRangeRequest → http_util.Request()
→ Response headers available in streaming callback but not in final response object passed to LogRequest ✗

**Impact:**

- request.headers['Range'] is logged correctly
- response.headers['Content-Range'] is NULL in logs


**Suggested fix:**

Ensure response headers from the streaming on_response callback in GetRangeRequest are preserved in the HTTPResponse object returned by http_util.Request(), so LogRequest can log them.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in httpfs_httplib_client.cpp and trace GetRangeRequest through http_util.Request() to LogRequest, comparing the streaming on_response callback with the final HTTPResponse. Reproduce the cache_httpfs path with HTTP logging and verify that response.headers['Content-Range'] is populated while request.headers['Range'] remains logged.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
networking, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.