duckdb / duckdb/duckdb-httpfs

ParseCommonPrefix crashes when S3 backend adds xmlns attributes on inner XML elements (e.g. Tigris)

Open
#281 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

## Bug

`FindTagContents` matches `` exactly, so it fails on ``. This causes `ParseCommonPrefix` to throw `InternalException("Parsing error while parsing s3 listobject result")` when using S3-compatible backends that add XML namespace attributes on inner elements.

## Reproduction

Tigris (t3.storage.dev) returns this from ListObjectsV2 with `delimiter=/`:

```xml


path/to/prefix/

```

AWS S3 and Cloudflare R2 only set `xmlns` on the root element. The Tigris response is valid XML (redundant namespace inheritance), but `FindTagContents` does `find("<" + tag + ">")` which doesn't match ``.

Any glob pattern that triggers `delimiter=/` listing (i.e. `*` without `**`, with `s3_allow_recursive_globbing = true`) will crash against Tigris.

**Workaround:** `SET s3_allow_recursive_globbing = false`

## Root cause

`FindTagContents` (line ~1190) constructs `"<" + tag + ">"` and does an exact `string::find`. The `>` prevents matching elements with attributes.

`ParseFileList` was migrated to `FindTagContents` in the "faster globbing" work (PR #218), but `ParseCommonPrefix` and `ParseContinuationToken` still used raw `find("")` with hardcoded offsets (`+8`, `+23`). All three share the same underlying vulnerability in `FindTagContents`.

## Suggested fix

We have a working fix with tests on our fork: [supersensory-technologies/duckdb-httpfs@fix/xml-namespace-parsing](https://github.com/supersensory-technologies/duckdb-httpfs/compare/main...fix/xml-namespace-parsing)

Two changes:
1. **`FindTagContents`**: match ``), verify next char is `>` or ` `, then scan to `>` for content start
2. **`ParseCommonPrefix` + `ParseContinuationToken`**: refactor to use `FindTagContents` (consistent with `ParseFileList`)

8 unit tests (Catch2) covering AWS-style, Tigris-style, multiple entries, and empty responses — all passing. Also verified against live Tigris endpoint.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.