tutorials/azureml-getting-started (credits-default-model) fails to deploy with: "Expected 96 from C header, got 88 from PyObject"

### Operating System

MacOS

### Version Information

Python Version: 3.12.7
azure-ai-ml package version: 1.21.0

### Steps to reproduce

1. Load [azureml-getting-started-studio.ipynb](https://github.com/Azure/azureml-examples/blob/main/tutorials/azureml-getting-started/azureml-getting-started-studio.ipynb) into VS Code
2. Add the code block below before the first original block (otherwise, authentication fails)
```
from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential

try:
credential = DefaultAzureCredential()
# Check if given credential can get token successfully.
credential.get_token("https://management.azure.com/.default")
except Exception as ex:
# Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work
# This will open a browser page for
credential = InteractiveBrowserCredential()
```

3. Run everything successfully until step/item 7 (_Deploy the model as an online endpoint_)
4. Follow the _Endpoint creation & deployment via wizard UI_ instructions, receive a **Failed** under the deployment's _Provisioning State_ and the error log below:

```
Instance status:
SystemSetup: Succeeded
UserContainerImagePull: Succeeded
ModelDownload: Succeeded
UserContainerStart: InProgress

Container events:
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:12.478572Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:22.494146Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:32.498042Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:42.497885Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:52.478491Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:02.494329Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:12.494084Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:22.494119Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:32.494126Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:42.497857Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:52.494157Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:02.49442Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:12.494019Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:22.494261Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:32.494194Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:42.49788Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:52.494265Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:06:02.49823Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:06:12.494149Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:06:22.494347Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502

Container logs:
2024-10-10T11:00:05,636783121+00:00 - rsyslog/run
2024-10-10T11:00:05,660534689+00:00 - nginx/run
2024-10-10T11:00:05,666009973+00:00 - gunicorn/run
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1
2024-10-10T11:00:05,699105686+00:00 | gunicorn/run |
2024-10-10T11:00:05,706316197+00:00 | gunicorn/run | ###############################################
2024-10-10T11:00:05,710913868+00:00 | gunicorn/run | AzureML Container Runtime Information
2024-10-10T11:00:05,717056363+00:00 | gunicorn/run | ###############################################
2024-10-10T11:00:05,720074210+00:00 | gunicorn/run |
2024-10-10T11:00:05,723216459+00:00 | gunicorn/run |
2024-10-10T11:00:05,729737860+00:00 | gunicorn/run | AzureML image information: mlflow-ubuntu20.04-py38-cpu-inference:20240805.v5
2024-10-10T11:00:05,732576004+00:00 | gunicorn/run |
2024-10-10T11:00:05,735530049+00:00 | gunicorn/run |
2024-10-10T11:00:05,738387494+00:00 | gunicorn/run | PATH environment variable: /opt/miniconda/envs/amlenv/bin:/opt/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-10-10T11:00:05,741136636+00:00 | gunicorn/run | PYTHONPATH environment variable:
2024-10-10T11:00:05,795602279+00:00 | gunicorn/run |
2024-10-10T11:00:11,635188772+00:00 | gunicorn/run | CONDAPATH environment variable: /opt/miniconda

# conda environments:
#
base /opt/miniconda
amlenv /opt/miniconda/envs/amlenv

2024-10-10T11:00:14,920278755+00:00 | gunicorn/run |
2024-10-10T11:00:14,925375932+00:00 | gunicorn/run | Pip Dependencies (before dynamic installation)

annotated-types==0.7.0
azure-core==1.30.2
azure-identity==1.17.1
azureml-inference-server-http==1.3.0
blinker==1.8.2
cachetools==5.4.0
certifi==2024.7.4
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cryptography==43.0.0
Flask==2.3.2
Flask-Cors==3.0.10
google-api-core==2.19.1
google-auth==2.32.0
googleapis-common-protos==1.63.2
gunicorn==22.0.0
idna==3.7
importlib_metadata==8.2.0
inference-schema==1.8
itsdangerous==2.2.0
Jinja2==3.1.4
MarkupSafe==2.1.5
msal==1.30.0
msal-extensions==1.2.0
opencensus==0.11.4
opencensus-context==0.1.3
opencensus-ext-azure==1.1.13
packaging==24.1
portalocker==2.10.1
proto-plus==1.24.0
protobuf==5.27.3
psutil==6.0.0
pyasn1==0.6.0
pyasn1_modules==0.4.0
pycparser==2.22
pydantic==2.7.4
pydantic-settings==2.4.0
pydantic_core==2.18.4
PyJWT==2.9.0
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
pytz==2024.1
requests==2.32.3
rsa==4.9
six==1.16.0
typing_extensions==4.12.2
urllib3==2.2.2
Werkzeug==3.0.3
wrapt==1.16.0
zipp==3.19.2

2024-10-10T11:00:20,606892801+00:00 | gunicorn/run |
2024-10-10T11:00:20,610213452+00:00 | gunicorn/run | Entry script directory: /var/mlflow_resources/.
2024-10-10T11:00:20,613244298+00:00 | gunicorn/run |
2024-10-10T11:00:20,618317077+00:00 | gunicorn/run | ###############################################
2024-10-10T11:00:20,621522426+00:00 | gunicorn/run | Dynamic Python Package Installation
2024-10-10T11:00:20,624719075+00:00 | gunicorn/run | ###############################################
2024-10-10T11:00:20,627784922+00:00 | gunicorn/run |
2024-10-10T11:00:20,631190575+00:00 | gunicorn/run | Updating conda environment from /var/azureml-app/azureml-models/credit_defaults_model/1/credit_defaults_model/conda.yaml !
Retrieving notices: ...working... done
Channels:
- conda-forge
- defaults
Platform: linux-64
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

==> WARNING: A newer version of conda exists. <==
current version: 24.7.1
latest version: 24.9.1

Please update conda by running

$ conda update -n base -c conda-forge conda

Downloading and Extracting Packages: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/opt/miniconda/envs/userenv/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/condaenv.v_2c6wn8.requirements.txt', '--exists-action=b']
Pip subprocess output:
Collecting mlflow==2.16.2
Downloading mlflow-2.16.2-py3-none-any.whl (26.7 MB)
Collecting backports-tempfile==1.0
Downloading backports.tempfile-1.0-py2.py3-none-any.whl (4.4 kB)
Collecting cloudpickle==2.2.1
Downloading cloudpickle-2.2.1-py3-none-any.whl (25 kB)
Collecting jaraco-collections==5.1.0
Downloading jaraco.collections-5.1.0-py3-none-any.whl (11 kB)
Collecting pandas==1.5.3
Downloading pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.1 MB)
Collecting psutil==5.8.0
Downloading psutil-5.8.0.tar.gz (470 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting scikit-learn==1.5.2
Downloading scikit_learn-1.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.3 MB)
Collecting graphene<4
Downloading graphene-3.3-py2.py3-none-any.whl (128 kB)
Collecting sqlalchemy<3,>=1.4.0
Downloading SQLAlchemy-2.0.35-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Collecting alembic!=1.10.0,<2
Downloading alembic-1.13.3-py3-none-any.whl (233 kB)
Collecting markdown<4,>=3.3
Downloading Markdown-3.7-py3-none-any.whl (106 kB)
Collecting matplotlib<4
Downloading matplotlib-3.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB)
Collecting Flask<4
Downloading flask-3.0.3-py3-none-any.whl (101 kB)
Collecting mlflow-skinny==2.16.2
Downloading mlflow_skinny-2.16.2-py3-none-any.whl (5.6 MB)
Collecting numpy<3
Downloading numpy-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.3 MB)
Collecting Jinja2<4,>=2.11
Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
Collecting gunicorn<24
Downloading gunicorn-23.0.0-py3-none-any.whl (85 kB)
Collecting pyarrow<18,>=4.0.0
Downloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (39.9 MB)
Collecting docker<8,>=4.0.0
Downloading docker-7.1.0-py3-none-any.whl (147 kB)
Collecting scipy<2
Downloading scipy-1.14.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (41.2 MB)
Collecting backports.weakref
Downloading backports.weakref-1.0.post1-py2.py3-none-any.whl (5.2 kB)
Collecting jaraco.text
Downloading jaraco.text-4.0.0-py3-none-any.whl (11 kB)
Collecting python-dateutil>=2.8.1
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Collecting pytz>=2020.1
Downloading pytz-2024.2-py2.py3-none-any.whl (508 kB)
Collecting joblib>=1.2.0
Downloading joblib-1.4.2-py3-none-any.whl (301 kB)
Collecting threadpoolctl>=3.1.0
Downloading threadpoolctl-3.5.0-py3-none-any.whl (18 kB)
Collecting sqlparse<1,>=0.4.0
Downloading sqlparse-0.5.1-py3-none-any.whl (44 kB)
Collecting opentelemetry-api<3,>=1.9.0
Downloading opentelemetry_api-1.27.0-py3-none-any.whl (63 kB)
Collecting click<9,>=7.0
Downloading click-8.1.7-py3-none-any.whl (97 kB)
Collecting gitpython<4,>=3.1.9
Downloading GitPython-3.1.43-py3-none-any.whl (207 kB)
Collecting importlib-metadata!=4.7.0,<9,>=3.7.0
Downloading importlib_metadata-8.5.0-py3-none-any.whl (26 kB)
Collecting pyyaml<7,>=5.1
Downloading PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (751 kB)
Collecting cachetools<6,>=5.0.0
Downloading cachetools-5.5.0-py3-none-any.whl (9.5 kB)
Collecting databricks-sdk<1,>=0.20.0
Downloading databricks_sdk-0.34.0-py3-none-any.whl (565 kB)
Collecting opentelemetry-sdk<3,>=1.9.0
Downloading opentelemetry_sdk-1.27.0-py3-none-any.whl (110 kB)
Collecting requests<3,>=2.17.3
Downloading requests-2.32.3-py3-none-any.whl (64 kB)
Collecting protobuf<6,>=3.12.0
Downloading protobuf-5.28.2-cp38-abi3-manylinux2014_x86_64.whl (316 kB)
Collecting packaging<25
Downloading packaging-24.1-py3-none-any.whl (53 kB)
Collecting Mako
Downloading Mako-1.3.5-py3-none-any.whl (78 kB)
Collecting typing-extensions>=4
Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Collecting urllib3>=1.26.0
Downloading urllib3-2.2.3-py3-none-any.whl (126 kB)
Collecting itsdangerous>=2.1.2
Downloading itsdangerous-2.2.0-py3-none-any.whl (16 kB)
Collecting Werkzeug>=3.0.0
Downloading werkzeug-3.0.4-py3-none-any.whl (227 kB)
Collecting blinker>=1.6.2
Downloading blinker-1.8.2-py3-none-any.whl (9.5 kB)
Collecting graphql-relay<3.3,>=3.1
Downloading graphql_relay-3.2.0-py3-none-any.whl (16 kB)
Collecting aniso8601<10,>=8
Downloading aniso8601-9.0.1-py2.py3-none-any.whl (52 kB)
Collecting graphql-core<3.3,>=3.1
Downloading graphql_core-3.2.4-py3-none-any.whl (203 kB)
Collecting MarkupSafe>=2.0
Downloading MarkupSafe-3.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (20 kB)
Collecting pyparsing>=2.3.1
Downloading pyparsing-3.1.4-py3-none-any.whl (104 kB)
Collecting fonttools>=4.22.0
Downloading fonttools-4.54.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
Collecting cycler>=0.10
Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB)
Collecting kiwisolver>=1.3.1
Downloading kiwisolver-1.4.7-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
Collecting pillow>=8
Downloading pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.5 MB)
Collecting contourpy>=1.0.1
Downloading contourpy-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (322 kB)
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting greenlet!=0.4.17
Downloading greenlet-3.1.1-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (599 kB)
Collecting jaraco.functools
Downloading jaraco.functools-4.1.0-py3-none-any.whl (10 kB)
Collecting more-itertools
Downloading more_itertools-10.5.0-py3-none-any.whl (60 kB)
Collecting autocommand
Downloading autocommand-2.2.2-py3-none-any.whl (19 kB)
Collecting jaraco.context>=4.1
Downloading jaraco.context-6.0.1-py3-none-any.whl (6.8 kB)
Collecting google-auth~=2.0
Downloading google_auth-2.35.0-py2.py3-none-any.whl (208 kB)
Collecting gitdb<5,>=4.0.1
Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)
Collecting zipp>=3.20
Downloading zipp-3.20.2-py3-none-any.whl (9.2 kB)
Collecting backports.tarfile
Downloading backports.tarfile-1.2.0-py3-none-any.whl (30 kB)
Collecting importlib-metadata!=4.7.0,<9,>=3.7.0
Downloading importlib_metadata-8.4.0-py3-none-any.whl (26 kB)
Collecting deprecated>=1.2.6
Downloading Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB)
Collecting opentelemetry-semantic-conventions==0.48b0
Downloading opentelemetry_semantic_conventions-0.48b0-py3-none-any.whl (149 kB)
Collecting charset-normalizer<4,>=2
Downloading charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)
Collecting certifi>=2017.4.17
Downloading certifi-2024.8.30-py3-none-any.whl (167 kB)
Collecting idna<4,>=2.5
Downloading idna-3.10-py3-none-any.whl (70 kB)
Collecting wrapt<2,>=1.10
Downloading wrapt-1.16.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB)
Collecting smmap<6,>=3.0.1
Downloading smmap-5.0.1-py3-none-any.whl (24 kB)
Collecting rsa<5,>=3.1.4
Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
Downloading pyasn1_modules-0.4.1-py3-none-any.whl (181 kB)
Collecting pyasn1<0.7.0,>=0.4.6
Downloading pyasn1-0.6.1-py3-none-any.whl (83 kB)
Building wheels for collected packages: psutil
Building wheel for psutil (setup.py): started
Building wheel for psutil (setup.py): finished with status 'done'
Created wheel for psutil: filename=psutil-5.8.0-cp310-cp310-linux_x86_64.whl size=229611 sha256=72ee154aa9763cd0ed84a9a5f203ae36bf1a6e13ecb7d149cdc2a8f3c86f8a37
Stored in directory: /home/dockeruser/.cache/pip/wheels/12/a3/6d/615295409067d58a62a069d30d296d61d3ac132605e3a9555c
Successfully built psutil
Installing collected packages: zipp, wrapt, pyasn1, importlib-metadata, deprecated, urllib3, smmap, rsa, pyasn1-modules, opentelemetry-api, idna, charset-normalizer, certifi, cachetools, typing-extensions, six, requests, opentelemetry-semantic-conventions, numpy, more-itertools, MarkupSafe, greenlet, graphql-core, google-auth, gitdb, backports.tarfile, Werkzeug, threadpoolctl, sqlparse, sqlalchemy, scipy, pyyaml, pytz, python-dateutil, pyparsing, protobuf, pillow, packaging, opentelemetry-sdk, Mako, kiwisolver, joblib, Jinja2, jaraco.functools, jaraco.context, itsdangerous, graphql-relay, gitpython, fonttools, databricks-sdk, cycler, contourpy, cloudpickle, click, blinker, autocommand, aniso8601, scikit-learn, pyarrow, pandas, mlflow-skinny, matplotlib, markdown, jaraco.text, gunicorn, graphene, Flask, docker, backports.weakref, alembic, psutil, mlflow, jaraco-collections, backports-tempfile
Successfully installed Flask-3.0.3 Jinja2-3.1.4 Mako-1.3.5 MarkupSafe-3.0.1 Werkzeug-3.0.4 alembic-1.13.3 aniso8601-9.0.1 autocommand-2.2.2 backports-tempfile-1.0 backports.tarfile-1.2.0 backports.weakref-1.0.post1 blinker-1.8.2 cachetools-5.5.0 certifi-2024.8.30 charset-normalizer-3.4.0 click-8.1.7 cloudpickle-2.2.1 contourpy-1.3.0 cycler-0.12.1 databricks-sdk-0.34.0 deprecated-1.2.14 docker-7.1.0 fonttools-4.54.1 gitdb-4.0.11 gitpython-3.1.43 google-auth-2.35.0 graphene-3.3 graphql-core-3.2.4 graphql-relay-3.2.0 greenlet-3.1.1 gunicorn-23.0.0 idna-3.10 importlib-metadata-8.4.0 itsdangerous-2.2.0 jaraco-collections-5.1.0 jaraco.context-6.0.1 jaraco.functools-4.1.0 jaraco.text-4.0.0 joblib-1.4.2 kiwisolver-1.4.7 markdown-3.7 matplotlib-3.9.2 mlflow-2.16.2 mlflow-skinny-2.16.2 more-itertools-10.5.0 numpy-2.1.2 opentelemetry-api-1.27.0 opentelemetry-sdk-1.27.0 opentelemetry-semantic-conventions-0.48b0 packaging-24.1 pandas-1.5.3 pillow-10.4.0 protobuf-5.28.2 psutil-5.8.0 pyarrow-17.0.0 pyasn1-0.6.1 pyasn1-modules-0.4.1 pyparsing-3.1.4 python-dateutil-2.9.0.post0 pytz-2024.2 pyyaml-6.0.2 requests-2.32.3 rsa-4.9 scikit-learn-1.5.2 scipy-1.14.1 six-1.16.0 smmap-5.0.1 sqlalchemy-2.0.35 sqlparse-0.5.1 threadpoolctl-3.5.0 typing-extensions-4.12.2 urllib3-2.2.3 wrapt-1.16.0 zipp-3.20.2

done
#
# To activate this environment, use
#
# $ conda activate userenv
#
# To deactivate an active environment, use
#
# $ conda deactivate

WARNING: Package(s) not found: azureml-inference-server-http
2024-10-10T11:05:40,620470267+00:00 | gunicorn/run | Install azureml-inference-server-http version 1.3.0
Collecting azureml-inference-server-http==1.3.0
Downloading azureml_inference_server_http-1.3.0-py3-none-any.whl (42 kB)
Collecting inference-schema~=1.8.0
Downloading inference_schema-1.8-py3-none-any.whl (21 kB)
Collecting flask-cors~=3.0.1
Downloading Flask_Cors-3.0.10-py2.py3-none-any.whl (14 kB)
Collecting flask<=2.3.2
Downloading Flask-2.3.2-py3-none-any.whl (96 kB)
Collecting opencensus-ext-azure~=1.1.0
Downloading opencensus_ext_azure-1.1.13-py2.py3-none-any.whl (43 kB)
Requirement already satisfied: werkzeug>=3.0.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from azureml-inference-server-http==1.3.0) (3.0.4)
Collecting pydantic-settings
Downloading pydantic_settings-2.5.2-py3-none-any.whl (26 kB)
Collecting gunicorn==22.0.0
Downloading gunicorn-22.0.0-py3-none-any.whl (84 kB)
Collecting pydantic~=2.7.1
Downloading pydantic-2.7.4-py3-none-any.whl (409 kB)
Requirement already satisfied: packaging in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from gunicorn==22.0.0->azureml-inference-server-http==1.3.0) (24.1)
Requirement already satisfied: click>=8.1.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (8.1.7)
Requirement already satisfied: itsdangerous>=2.1.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (2.2.0)
Requirement already satisfied: blinker>=1.6.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (1.8.2)
Requirement already satisfied: Jinja2>=3.1.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (3.1.4)
Requirement already satisfied: Six in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask-cors~=3.0.1->azureml-inference-server-http==1.3.0) (1.16.0)
Requirement already satisfied: python-dateutil>=2.5.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from inference-schema~=1.8.0->azureml-inference-server-http==1.3.0) (2.9.0.post0)
Requirement already satisfied: wrapt<=1.16.0,>=1.14.0 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from inference-schema~=1.8.0->azureml-inference-server-http==1.3.0) (1.16.0)
Requirement already satisfied: pytz>=2017.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from inference-schema~=1.8.0->azureml-inference-server-http==1.3.0) (2024.2)
Collecting azure-core<2.0.0,>=1.12.0
Downloading azure_core-1.31.0-py3-none-any.whl (197 kB)
Requirement already satisfied: requests>=2.19.0 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2.32.3)
Collecting azure-identity<2.0.0,>=1.5.0
Downloading azure_identity-1.19.0-py3-none-any.whl (187 kB)
Collecting opencensus<1.0.0,>=0.11.4
Downloading opencensus-0.11.4-py2.py3-none-any.whl (128 kB)
Requirement already satisfied: psutil>=5.6.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (5.8.0)
Collecting pydantic-core==2.18.4
Downloading pydantic_core-2.18.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)
Collecting annotated-types>=0.4.0
Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB)
Requirement already satisfied: typing-extensions>=4.6.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from pydantic~=2.7.1->azureml-inference-server-http==1.3.0) (4.12.2)
Requirement already satisfied: MarkupSafe>=2.1.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from werkzeug>=3.0.3->azureml-inference-server-http==1.3.0) (3.0.1)
Collecting python-dotenv>=0.21.0
Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Collecting msal-extensions>=1.2.0
Downloading msal_extensions-1.2.0-py3-none-any.whl (19 kB)
Collecting cryptography>=2.5
Downloading cryptography-43.0.1-cp39-abi3-manylinux_2_28_x86_64.whl (4.0 MB)
Collecting msal>=1.30.0
Downloading msal-1.31.0-py3-none-any.whl (113 kB)
Collecting opencensus-context>=0.1.3
Downloading opencensus_context-0.1.3-py2.py3-none-any.whl (5.1 kB)
Collecting google-api-core<3.0.0,>=1.0.0
Downloading google_api_core-2.21.0-py3-none-any.whl (156 kB)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2.2.3)
Requirement already satisfied: idna<4,>=2.5 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2024.8.30)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (3.4.0)
Collecting cffi>=1.12
Downloading cffi-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (446 kB)
Collecting proto-plus<2.0.0dev,>=1.22.3
Downloading proto_plus-1.24.0-py3-none-any.whl (50 kB)
Requirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0.dev0,>=3.19.5 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (5.28.2)
Requirement already satisfied: google-auth<3.0.dev0,>=2.14.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2.35.0)
Collecting googleapis-common-protos<2.0.dev0,>=1.56.2
Downloading googleapis_common_protos-1.65.0-py2.py3-none-any.whl (220 kB)
Collecting PyJWT[crypto]<3,>=1.0.0
Downloading PyJWT-2.9.0-py3-none-any.whl (22 kB)
Collecting portalocker<3,>=1.4
Downloading portalocker-2.10.1-py3-none-any.whl (18 kB)
Collecting pycparser
Downloading pycparser-2.22-py3-none-any.whl (117 kB)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (5.5.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (4.9)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (0.4.1)
Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (0.6.1)
Installing collected packages: pycparser, cffi, PyJWT, cryptography, proto-plus, portalocker, msal, googleapis-common-protos, pydantic-core, opencensus-context, msal-extensions, google-api-core, azure-core, annotated-types, python-dotenv, pydantic, opencensus, flask, azure-identity, pydantic-settings, opencensus-ext-azure, inference-schema, gunicorn, flask-cors, azureml-inference-server-http
Attempting uninstall: flask
Found existing installation: flask 3.0.3
Uninstalling flask-3.0.3:
Successfully uninstalled flask-3.0.3
Attempting uninstall: gunicorn
Found existing installation: gunicorn 23.0.0
Uninstalling gunicorn-23.0.0:
Successfully uninstalled gunicorn-23.0.0
Successfully installed PyJWT-2.9.0 annotated-types-0.7.0 azure-core-1.31.0 azure-identity-1.19.0 azureml-inference-server-http-1.3.0 cffi-1.17.1 cryptography-43.0.1 flask-2.3.2 flask-cors-3.0.10 google-api-core-2.21.0 googleapis-common-protos-1.65.0 gunicorn-22.0.0 inference-schema-1.8 msal-1.31.0 msal-extensions-1.2.0 opencensus-0.11.4 opencensus-context-0.1.3 opencensus-ext-azure-1.1.13 portalocker-2.10.1 proto-plus-1.24.0 pycparser-2.22 pydantic-2.7.4 pydantic-core-2.18.4 pydantic-settings-2.5.2 python-dotenv-1.0.1
2024-10-10T11:06:18,005275637+00:00 | gunicorn/run |
2024-10-10T11:06:18,008905681+00:00 | gunicorn/run | ###############################################
2024-10-10T11:06:18,012396323+00:00 | gunicorn/run | Checking if the Python package azureml-inference-server-http is installed
2024-10-10T11:06:18,015871665+00:00 | gunicorn/run | ###############################################
2024-10-10T11:06:18,019106304+00:00 | gunicorn/run |
2024-10-10T11:06:22,619276944+00:00 | gunicorn/run |
2024-10-10T11:06:22,622712786+00:00 | gunicorn/run | ###############################################
2024-10-10T11:06:22,625700722+00:00 | gunicorn/run | AzureML Inference Server
2024-10-10T11:06:22,628489655+00:00 | gunicorn/run | ###############################################
2024-10-10T11:06:22,631337490+00:00 | gunicorn/run |
2024-10-10T11:06:22,634345926+00:00 | gunicorn/run | Starting AzureML Inference Server HTTP.
2024-10-10 11:06:23,311 I [12] azmlinfsrv - Loaded logging config from /opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/logging.json
2024-10-10 11:06:23,408 I [12] gunicorn.error - Starting gunicorn 22.0.0
2024-10-10 11:06:23,410 I [12] gunicorn.error - Listening at: http://0.0.0.0:31311 (12)
2024-10-10 11:06:23,410 I [12] gunicorn.error - Using worker: sync
2024-10-10 11:06:23,414 I [152] gunicorn.error - Booting worker with pid: 152
/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pydantic/_internal/_fields.py:160: UserWarning: Field "model_dc_storage_enabled" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ('settings_',)`.
warnings.warn(

Azure ML Inferencing HTTP server v1.3.0

Server Settings
---------------
Entry Script Name: /var/mlflow_resources/mlflow_score_script.py
Model Directory: /var/azureml-app/azureml-models/credit_defaults_model/1
Config File: None
Worker Count: 1
Worker Timeout (seconds): 300
Server Port: 31311
Health Port: 31311
Application Insights Enabled: false
Application Insights Key: None
Inferencing HTTP server version: azmlinfsrv/1.3.0
CORS for the specified origins: None
Create dedicated endpoint for health: None

Server Routes
---------------
Liveness Probe: GET 127.0.0.1:31311/
Score: POST 127.0.0.1:31311/score

Warnings
---------------
Azmlinfsrv will be migrating to Pydantic 2.0 on 1/15/24. This is a breaking change for any Pydantic 1.0 code.

2024-10-10 11:06:23,996 W [152] azmlinfsrv - Found extra keys in the config file that are not supported by the server.
Extra keys = ['AZUREML_ENTRY_SCRIPT', 'AZUREML_MODEL_DIR', 'HOSTNAME']
2024-10-10 11:06:25,124 I [152] azmlinfsrv - AML_FLASK_ONE_COMPATIBILITY is set. Patched Flask to ensure compatibility with Flask 1.
Initializing logger
2024-10-10 11:06:25,127 I [152] azmlinfsrv - Starting up app insights client
2024-10-10 11:06:26,402 E [152] azmlinfsrv - Traceback (most recent call last):
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/server/user_script.py", line 77, in load_script
main_module_spec.loader.exec_module(user_module)
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/var/mlflow_resources/mlflow_score_script.py", line 5, in
import mlflow
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/__init__.py", line 34, in
from mlflow import (
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/data/__init__.py", line 5, in
from mlflow.data import dataset_registry
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/data/dataset_registry.py", line 137, in
from mlflow.data.pandas_dataset import from_pandas
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/data/pandas_dataset.py", line 6, in
import pandas as pd
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/__init__.py", line 22, in
from pandas.compat import is_numpy_dev as _is_numpy_dev # pyright: ignore # noqa:F401
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/compat/__init__.py", line 18, in
from pandas.compat.numpy import (
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/compat/numpy/__init__.py", line 4, in
from pandas.util.version import Version
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/util/__init__.py", line 2, in
from pandas.util._decorators import ( # noqa:F401
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/util/_decorators.py", line 14, in
from pandas._libs.properties import cache_readonly
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/_libs/__init__.py", line 13, in
from pandas._libs.interval import Interval
File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/server/aml_blueprint.py", line 91, in setup
self.user_script.load_script(config.app_root)
File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/server/user_script.py", line 79, in load_script
raise UserScriptImportException(ex) from ex
azureml_inference_server_http.server.user_script.UserScriptImportException: Failed to import user script because it raised an unhandled exception

2024-10-10 11:06:26,404 I [152] gunicorn.error - Worker exiting (pid: 152)
2024-10-10 11:06:27,130 E [12] gunicorn.error - Worker (pid:152) exited with code 3
2024-10-10 11:06:27,132 E [12] gunicorn.error - Shutting down: Master
2024-10-10 11:06:27,133 E [12] gunicorn.error - Reason: Worker failed to boot.

Azure ML Inferencing HTTP server v1.3.0

Server Settings
---------------
Entry Script Name: /var/mlflow_resources/mlflow_score_script.py
Model Directory: /var/azureml-app/azureml-models/credit_defaults_model/1
Config File: None
Worker Count: 1
Worker Timeout (seconds): 300
Server Port: 31311
Health Port: 31311
Application Insights Enabled: false
Application Insights Key: None
Inferencing HTTP server version: azmlinfsrv/1.3.0
CORS for the specified origins: None
Create dedicated endpoint for health: None

Server Routes
---------------
Liveness Probe: GET 127.0.0.1:31311/
Score: POST 127.0.0.1:31311/score

Warnings
---------------
Azmlinfsrv will be migrating to Pydantic 2.0 on 1/15/24. This is a breaking change for any Pydantic 1.0 code.

2024-10-10T11:06:27,400529401+00:00 - gunicorn/finish 3 0
2024-10-10T11:06:27,404048457+00:00 - Exit code 3 is not normal. Killing image.

```

### Expected behavior

The endpoint is provisioned successfully.

### Actual behavior

Provisioning State = Failed

### Addition information

According to [this](https://github.com/pandas-dev/pandas/issues/59270) reference, the underlying error condition showed in the logs (**ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject**) is due to an incompatibility between the _numpy_ and _pandas_ package versions. Could this discrepancy be coming from the environment image (i.e. `environment="azureml://registries/azureml/environments/sklearn-1.5/labels/latest"`)? Or is it somehow related to my local environment?

Start with tutorials/azureml-getting-started/azureml-getting-started-studio.ipynb and reproduce step 7 using the reported Python 3.12.7 and azure-ai-ml 1.21.0 versions. Review the deployment and container logs around the credits-default-model environment and verify that the online endpoint deployment completes successfully.

Written by the indexing model from the issue text.