[Bug]: Memory Leak on Repeated /md Requests via Docker (MacOS) — Container Crashes Randomly Over Time

crawl4ai version

0.6.0-r2 or Latest docker image

Expected Behavior

When sending repeated POST requests to /md every 10 seconds with dynamic URLs, the Docker container should handle each request efficiently without memory leaks or crashes, maintaining stable memory usage over time.

Current Behavior

Initially, everything works fine. However, after some time (randomly, depending on duration and frequency), memory usage starts increasing significantly until the container crashes. This suggests a possible memory leak or improper cleanup of resources on repeated executions.

Attached below is a screenshot showing the memory spike before the crash.

Is this reproducible?

Yes

Inputs Causing the Bug

Example request payload sent via Postman every 10 seconds:

{
  "url": "http://www.example.com",
  "f": "fit",
  "q": null,
  "c": "0"
}

Steps to Reproduce

1.	Run the crawl4ai app using the Docker setup.
2.	Use a tool (like Postman or a script) to send POST requests to http://localhost/md every 10 seconds.
3.	Use a changing URL in the request payload for each call.
4.	Monitor Docker memory usage over time.
5.	Observe the steady increase in memory and eventual container crash.

Code snippets

OS

macOS Sequoia Version 15.5

Python version

3.12

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

Start with the Docker setup and reproduce the issue by sending changing-URL POST requests to /md every 10 seconds while monitoring container memory. Profile the repeated requests to identify what is not being cleaned up; done means memory remains stable and the container no longer crashes over sustained requests.