Airflow scheduler memory spikes when a dag is run with large run config

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

3.2.0

### What happened and how to reproduce it?

When a dag with dynamic task is triggered with large run config, the scheduler memory increases by 5-6 times the regular usage when scheduling the dynamic tasks. Once the dagrun completes, the memory usage is back to normal.

The run config is a json of size 512KB to 2MB

To reproduce, create a dag with dynamic task that create 500+ tasks. Trigger the dag with a json of size 512KB to 1MB as run config

### What you think should happen instead?

The scheduler memory should not spike when running dynamic task with large run config.

### Operating System

Debian GNU/Linux 12 (bookworm)

### Deployment

Other 3rd-party Helm chart

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.17.1
apache-airflow-providers-cncf-kubernetes==10.14.0
apache-airflow-providers-common-compat==1.14.1
apache-airflow-providers-common-io==1.7.1
apache-airflow-providers-common-sql==1.33.0
apache-airflow-providers-fab==3.6.1
apache-airflow-providers-smtp==2.4.3
apache-airflow-providers-standard==1.12.1

### Official Helm Chart version

Not Applicable

### Kubernetes Version

Not Applicable

### Helm Chart configuration

Not Applicable

### Docker Image customizations

Not Applicable

### Anything else?

Issue happens in dynamic dags, everytime, the dag is triggered with big config.

Older issue for reference - https://github.com/apache/airflow/issues/49076

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

No file or test is named. Start by reproducing the scheduler memory increase with a dynamic DAG creating 500+ tasks and a 512KB–1MB run config, then trace how the scheduler handles that configuration while scheduling tasks. Done means the scheduler no longer increases memory by 5–6 times during the run, while the reproduction remains functional.

Written by the indexing model from the issue text.