microsoft / microsoft/markitdown
Removing Real-time JavaScript in HTML Conversion Causes Loss of Dynamic Data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 186k
- Forks
- 13.7k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 49
Description
Description:
Markitdown's HTML converter wraps Markdownify, which removes all JavaScript and style blocks, including those responsible for loading real-time data. This issue affects documents requiring real-time data (e.g., live feeds, dynamic updates) that no longer appear in the HTML after conversion.
def convert(self, file_stream: BinaryIO, stream_info: StreamInfo, **kwargs: Any) -> DocumentConverterResult:
encoding = "utf-8" if stream_info.charset is None else stream_info.charset
soup = BeautifulSoup(file_stream, "html.parser", from_encoding=encoding)
# Remove javascript and style blocks
for script in soup(["script", "style"]):
script.extract()
Steps to Reproduce:
- Convert an HTML document containing JavaScript that loads real-time data (e.g., live updates or dynamic content).
- Observe that the real-time data is missing in the converted HTML output.
Expected Behavior:
The HTML converter should:
- Preserve JavaScript responsible for fetching or displaying real-time data.
- Allow configuration to selectively retain or remove JavaScript to prevent stripping dynamic content.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the HTML converter's convert method and inspect how BeautifulSoup removes script and style blocks. Determine how JavaScript retention should be configured without losing the existing removal behavior, then reproduce the conversion with a document containing dynamic data. Done means the requested dynamic content can be retained while JavaScript can still be selectively removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100