Why does `enable_cpu_mem_arena` have such a large effect on memory usage during inference?

Describe the bug
A clear and concise description of what the bug is. To avoid repetition please make sure this is not one of the known issues mentioned on the respective release page.

I'm performing inference using the Python API and a small ONNX model (~2MB) that was converted from a Keras .h5 model.

When running ort_sess.run() using default settings, memory usage skyrockets from ~200MB to ~6GB:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
   172    206.0 MiB    206.0 MiB           1   @profile
   173                                         def onnx_prediction(model_abs_path, input):
   174    206.0 MiB      0.0 MiB           1       sess_options = ort.SessionOptions()
   175    206.1 MiB      0.0 MiB           1       sess_options.enable_profiling = True
   176    212.6 MiB      6.6 MiB           1       ort_sess = ort.InferenceSession(model_abs_path, sess_opti
ons=sess_options)
   177   5792.0 MiB   5579.3 MiB           1       preds = ort_sess.run(output_names=["predictions"], input_
feed={"input_1": input})[0]
   178   5792.0 MiB      0.0 MiB           1       return preds

Searching in past GitHub issues, I found mention of enable_cpu_mem_arena. Setting this to False completely addresses the issue:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
   172    206.0 MiB    206.0 MiB           1   @profile
   173                                         def onnx_prediction(model_abs_path, input):
   174    206.1 MiB      0.0 MiB           1       sess_options = ort.SessionOptions()
   175    206.1 MiB      0.0 MiB           1       sess_options.enable_profiling = True
   176    206.1 MiB      0.0 MiB           1       sess_options.enable_cpu_mem_arena = False
   177    212.4 MiB      6.4 MiB           1       ort_sess = ort.InferenceSession(model_abs_path, sess_opti
ons=sess_options)
   178    217.8 MiB      5.3 MiB           1       preds = ort_sess.run(output_names=["predictions"], input_
feed={"input_1": input})[0]
   179    217.8 MiB      0.0 MiB           1       return preds

The docs on enable_cpu_mem_area mention:

Enables the memory arena on CPU. Arena may pre-allocate memory for future usage. Set this option to false if you don’t want it. Default is True.

But I have some questions to try to better understand what's actually going on here:

• Why was the CPU memory arena pre-allocating so much memory in the first place?
• Are there any risks or downsides to setting enable_cpu_mem_arena = False?

Urgency
If there are particular important use cases blocked by this or strict project-related timelines, please share more information and dates. If there are no hard deadlines, please specify none.

None.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
• ONNX Runtime installed from (source or binary): Binary
• ONNX Runtime version: 1.7.0
• Python version: 3.7
• Visual Studio version (if applicable): N/A
• GCC/Compiler version (if compiling from source): N/A
• CUDA/cuDNN version: N/A (CPU only)
• GPU model and memory: N/A (CPU only)
• CPU model: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz, 2801 Mhz, 4 Core(s), 8 Logical Processor(s)
• RAM: 16.0 GB

To Reproduce

• Describe steps/code to reproduce the behavior.
• Attach the ONNX model to the issue (where applicable) to expedite investigation.

from memory_profiler import profile

@profile
def onnx_prediction(model_path, input):
    ort_sess = ort.InferenceSession(model_path)
    preds = ort_sess.run(output_names=["predictions"], input_feed={"input_1": input})[0]
    return preds

Here is a .zip containing both an .onnx model file and a .npy array you can load to use for input: enable_cpu_memory_area_example.zip

Expected behavior
A clear and concise description of what you expected to happen.

Not pre-allocating 6GB of memory for a 2MB model.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here. If the issue is about a particular model, please share the model details as well to facilitate debugging.

Start by running the supplied Python reproduction with the attached ONNX model and .npy input, comparing default SessionOptions with enable_cpu_mem_arena=False. Inspect the CPU arena behavior behind SessionOptions and use the observed allocation pattern to explain the 6GB allocation and the trade-offs of disabling it; done means a documented explanation and clear guidance on the setting.