microsoft / microsoft/markitdown

Removing Real-time JavaScript in HTML Conversion Causes Loss of Dynamic Data

Open
#1,169 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
186k
Forks
13.7k
Avg merge
1d 4h
Merged PRs (30d)
49

Description

Description:
Markitdown's HTML converter wraps Markdownify, which removes all JavaScript and style blocks, including those responsible for loading real-time data. This issue affects documents requiring real-time data (e.g., live feeds, dynamic updates) that no longer appear in the HTML after conversion.

def convert(self, file_stream: BinaryIO, stream_info: StreamInfo, **kwargs: Any) -> DocumentConverterResult:
    encoding = "utf-8" if stream_info.charset is None else stream_info.charset
    soup = BeautifulSoup(file_stream, "html.parser", from_encoding=encoding)

    # Remove javascript and style blocks
    for script in soup(["script", "style"]):
        script.extract()

Steps to Reproduce:

  1. Convert an HTML document containing JavaScript that loads real-time data (e.g., live updates or dynamic content).
  2. Observe that the real-time data is missing in the converted HTML output.

Expected Behavior:
The HTML converter should:

  • Preserve JavaScript responsible for fetching or displaying real-time data.
  • Allow configuration to selectively retain or remove JavaScript to prevent stripping dynamic content.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the HTML converter's convert method and inspect how BeautifulSoup removes script and style blocks. Determine how JavaScript retention should be configured without losing the existing removal behavior, then reproduce the conversion with a document containing dynamic data. Done means the requested dynamic content can be retained while JavaScript can still be selectively removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.