openai-whisper-large batch endpoint, submit job failure: Failed building wheel for pycocotools

### Operating System

Linux

### Version Information

python version: 3.10
azure-ai-ml==1.9.0

### Steps to reproduce

1. Follow this tutorial: https://github.com/Azure/azureml-examples/blob/main/sdk/python/foundation-models/system/inference/automatic-speech-recognition/asr-batch-endpoint.ipynb
2. At step 5, "Run a batch inference job", the job failed with error "ERROR: Failed building wheel for pycocotools
ERROR: Could not build wheels for pycocotools, which is required to install pyproject.toml-based projects"

### Expected behavior

the job could run successfully

### Actual behavior

job failed

### Addition information

Full error
Building wheels for collected packages: wget, pycocotools, audioread, databricks-cli, fusepy, rouge-score, seqeval, future, lit
Building wheel for wget (setup.py): started
Building wheel for wget (setup.py): finished with status 'done'
Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9657 sha256=071432923f7fa0f344e58965dee563fade14b810bc2a1c6a6726914eb73b5fd8
Stored in directory: /root/.cache/pip/wheels/46/4a/43/6e71c9584e8b20b326931deaf89f11bf4db1bda170ac38e7b4
Building wheel for pycocotools (pyproject.toml): started
Building wheel for pycocotools (pyproject.toml): finished with status 'error'
Building wheel for audioread (setup.py): started
Building wheel for audioread (setup.py): finished with status 'done'
Created wheel for audioread: filename=audioread-3.0.0-py3-none-any.whl size=23703 sha256=b98cd616cd2952377f58e9d01dc541234b72859e3996ed77015d3f0765d14fa5
Stored in directory: /root/.cache/pip/wheels/74/25/48/ad94b69151b78e9aeba6850da119f04eda1c811d22fcf4b32d
Building wheel for databricks-cli (setup.py): started
Building wheel for databricks-cli (setup.py): finished with status 'done'
Created wheel for databricks-cli: filename=databricks_cli-0.17.7-py3-none-any.whl size=143860 sha256=37b05d8f08afa241de784149024f3e0fec5a2b617b14fd136ffbb49fa2097cab
Stored in directory: /root/.cache/pip/wheels/35/d3/b7/3c0e882c34e5809296ce94413a2555fb1fa5ba3d17da96f3c6
Building wheel for fusepy (setup.py): started
Building wheel for fusepy (setup.py): finished with status 'done'
Created wheel for fusepy: filename=fusepy-3.0.1-py3-none-any.whl size=10486 sha256=f71aeb82a1f9ca38b08291e7a2252726de03a7554edc50468f865c8e8b9c05b6
Stored in directory: /root/.cache/pip/wheels/db/01/ee/105147547cf0ecd1031b26901757e1f17c98902c8cc7bcaa47
Building wheel for rouge-score (setup.py): started
Building wheel for rouge-score (setup.py): finished with status 'done'
Created wheel for rouge-score: filename=rouge_score-0.1.2-py3-none-any.whl size=24936 sha256=cc8972bffa71c19cf1b84ef8caa9a23f4dcb23e9dcff54a678c236bd85b6a202
Stored in directory: /root/.cache/pip/wheels/5a/a8/47/66a086bcca0e429716cd68c4da6b7f9ac7976868735937eb63
Building wheel for seqeval (setup.py): started
Building wheel for seqeval (setup.py): finished with status 'done'
Created wheel for seqeval: filename=seqeval-1.2.2-py3-none-any.whl size=16165 sha256=c27dd16520ef0bac42384e6ba4f6cc2c88710976a24bc6f90bb16dd081eaa937
Stored in directory: /root/.cache/pip/wheels/e3/30/9b/6b670dac34775f2b7cc4e9b172202e81fbb4f9cdb103c1ca66
Building wheel for future (setup.py): started
Building wheel for future (setup.py): finished with status 'done'
Created wheel for future: filename=future-0.18.3-py3-none-any.whl size=492025 sha256=3bc99b21dadc4732c8c37ad8bc84a9ea6920c5948a41cb38a8cfc1b8f878eb59
Stored in directory: /root/.cache/pip/wheels/a6/db/41/71a0e5d071a14e716cc11bb021a9caa8f76ec337eca071487e
Building wheel for lit (pyproject.toml): started
Building wheel for lit (pyproject.toml): finished with status 'done'
Created wheel for lit: filename=lit-16.0.6-py3-none-any.whl size=93584 sha256=27c730695537db680b4090e97584f0438edfb220fd34bcb8bf25648d1cbaee01
Stored in directory: /root/.cache/pip/wheels/31/26/a6/958a0b82f632a115bc37080c7f3b04baaaa7f55f07d5ffa6c0
Successfully built wget audioread databricks-cli fusepy rouge-score seqeval future lit
Failed to build pycocotools

failed
Pip subprocess error:
error: subprocess-exited-with-error

× Building wheel for pycocotools (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [23 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-38
creating build/lib.linux-x86_64-cpython-38/pycocotools
copying pycocotools/coco.py -> build/lib.linux-x86_64-cpython-38/pycocotools
copying pycocotools/mask.py -> build/lib.linux-x86_64-cpython-38/pycocotools
copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-cpython-38/pycocotools
copying pycocotools/__init__.py -> build/lib.linux-x86_64-cpython-38/pycocotools
running build_ext
Compiling pycocotools/_mask.pyx because it changed.
[1/1] Cythonizing pycocotools/_mask.pyx
building 'pycocotools._mask' extension
creating build/common
creating build/temp.linux-x86_64-cpython-38
creating build/temp.linux-x86_64-cpython-38/common
creating build/temp.linux-x86_64-cpython-38/pycocotools
gcc -pthread -B /azureml-envs/azureml_6d341b69b79ddf697bca4a07cf2a1ac6/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tmp/pip-build-env-966dvlnc/overlay/lib/python3.8/site-packages/numpy/core/include -I./common -I/azureml-envs/azureml_6d341b69b79ddf697bca4a07cf2a1ac6/include/python3.8 -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-38/../common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
gcc: error: ../common/maskApi.c: No such file or directory
/tmp/pip-build-env-966dvlnc/overlay/lib/python3.8/site-packages/Cython/Compiler/Main.py:384: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-gr5kge21/pycocotools_111ec10f337f48e1b69eedd76c6c836d/pycocotools/_mask.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pycocotools
ERROR: Could not build wheels for pycocotools, which is required to install pyproject.toml-based projects

CondaEnvException: Pip failed

The command '/bin/sh -c ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_6d341b69b79ddf697bca4a07cf2a1ac6 -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name __pycache__ -exec rm -rf {} + && ldconfig' returned a non-zero code: 1
2023/08/27 21:33:22 Container failed during run: acb_step_0. No retries remaining.
failed to run step ID: acb_step_0: exit status 1

Run ID: crea failed after 3m15s. Error: failed during run, err: exit status 1

Start with the asr-batch-endpoint.ipynb tutorial, especially step 5, and inspect azureml-environment-setup/mutated_conda_dependencies.yml alongside the reported pycocotools build output. Reproduce the batch inference job and determine whether the environment setup completes; done means the job runs successfully without the pycocotools wheel failure.

Written by the indexing model from the issue text.