microsoft / microsoft/markitdown

RSS/Atom: urljoin collapses significant repeated slashes in resolved links

Open
#2,468 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
186k
Forks
13.7k
Avg merge
1d 4h
Merged PRs (30d)
49

Description

Summary

RSS/Atom link resolution can collapse significant repeated slashes in URL paths. For example, exports/2026//report.csv resolves to a URL ending in exports/2026/report.csv, potentially identifying a different resource.

This is a longstanding upstream limitation in Python's standard-library urllib.parse.urljoin(), newly exposed by the feed link-resolution changes in #2432. MarkItDown v0.1.7 preserved the original relative reference.

Disposition: deferred; not a blocker for the upcoming release. This issue tracks the limitation and the upstream fix rather than requesting custom URL-resolution code immediately.

Reproduction

No network access is needed. Reproduced on Python 3.12.3:

from io import BytesIO
from markitdown import MarkItDown, StreamInfo

feed = b'''<rss version="2.0"><channel>
<title>Exports</title><link>https://example.com/bucket/</link>
<description>Available reports</description>
<item><title>Report</title><description><![CDATA[
<a href="exports/2026//report.csv">Download</a>
]]></description></item>
</channel></rss>'''

result = MarkItDown().convert_stream(
    BytesIO(feed),
    stream_info=StreamInfo(
        extension=".rss",
        charset="utf-8",
        url="https://example.com/bucket/feed.xml",
    ),
)
print(result.markdown)

Actual link:

[Download](https://example.com/bucket/exports/2026/report.csv)

Expected link:

[Download](https://example.com/bucket/exports/2026//report.csv)

Cause and impact

_resolve_url() in packages/markitdown/src/markitdown/converters/_rss_converter.py calls urllib.parse.urljoin() directly. Its relative-path merge filters out empty interior segments.

Repeated slashes represent empty path segments and are not universally equivalent to a single slash. Some servers normalize them, but path-sensitive services and object stores can distinguish them. The same helper handles image references and xml:base; repeated slashes inherited from the base path can also be affected.

Upstream tracking

  • Python issue: python/cpython#84774
  • Proposed upstream fix: python/cpython#126679

As checked on September 11, 2026, both remain open and the filtering is still present in CPython's 3.13, 3.14, 3.15, and development main branches, as well as the local Python 3.12.3 runtime.

Because MarkItDown calls the standard library directly, it would inherit a future upstream fix when running on a Python release containing it. Updating MarkItDown alone would not change the behavior of an older Python runtime.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/markitdown/src/markitdown/converters/_rss_converter.py and inspect _resolve_url(), then reproduce the behavior with the supplied Python snippet. Review python/cpython#84774 and python/cpython#126679; done requires an upstream resolution and verifying that MarkItDown preserves repeated path slashes on supported Python versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.