ParseCommonPrefix crashes when S3 backend adds xmlns attributes on inner XML elements (e.g. Tigris)
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
## Bug
`FindTagContents` matches `` exactly, so it fails on ``. This causes `ParseCommonPrefix` to throw `InternalException("Parsing error while parsing s3 listobject result")` when using S3-compatible backends that add XML namespace attributes on inner elements.
## Reproduction
Tigris (t3.storage.dev) returns this from ListObjectsV2 with `delimiter=/`:
```xml
path/to/prefix/
```
AWS S3 and Cloudflare R2 only set `xmlns` on the root element. The Tigris response is valid XML (redundant namespace inheritance), but `FindTagContents` does `find("<" + tag + ">")` which doesn't match ``.
Any glob pattern that triggers `delimiter=/` listing (i.e. `*` without `**`, with `s3_allow_recursive_globbing = true`) will crash against Tigris.
**Workaround:** `SET s3_allow_recursive_globbing = false`
## Root cause
`FindTagContents` (line ~1190) constructs `"<" + tag + ">"` and does an exact `string::find`. The `>` prevents matching elements with attributes.
`ParseFileList` was migrated to `FindTagContents` in the "faster globbing" work (PR #218), but `ParseCommonPrefix` and `ParseContinuationToken` still used raw `find("")` with hardcoded offsets (`+8`, `+23`). All three share the same underlying vulnerability in `FindTagContents`.
## Suggested fix
We have a working fix with tests on our fork: [supersensory-technologies/duckdb-httpfs@fix/xml-namespace-parsing](https://github.com/supersensory-technologies/duckdb-httpfs/compare/main...fix/xml-namespace-parsing)
Two changes:
1. **`FindTagContents`**: match ``), verify next char is `>` or ` `, then scan to `>` for content start
2. **`ParseCommonPrefix` + `ParseContinuationToken`**: refactor to use `FindTagContents` (consistent with `ParseFileList`)
8 unit tests (Catch2) covering AWS-style, Tigris-style, multiple entries, and empty responses — all passing. Also verified against live Tigris endpoint.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.