Azure / Azure/azureml-examples

Example Pipeline with Azure OpenAI CommandComponents Fails to Import Data

Open
#2,941 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Operating System

Linux

### Version Information

Python Version: 3.10.11
azure-ai-ml package version: 1.8.0 (and also 1.12.1)

### Steps to reproduce

1. Create an Azure ML workspace
2. Inside the workspace, create a compute instance (I'm using a `Standard_DS11_v2` VM)
3. Turn on the instance, and access it locally from VSCode via a websocket (using the Azure ML extension)
4. Clone this repo
5. Find the credentials for your workspace
6. Run the code in the notebook

### Expected behavior

Code should create a new pipeline job in the Azure ML workspace, that finetunes a GPT 3.5 Turbo model using a user-defined dataset (stored in a directory in this repo).

### Actual behavior

The pipeline job is created, yet it fails.

It has 2 nodes - the failure is at the first, "Data Import"
Screenshot 2024-01-09 at 3 46 50 PM

The error at this node says: `UserError: Failed to submit job due to Exception: Response status code does not indicate success: 404 (Could not find datastore: azureml_managed_openaidevaulttrainingdata.).
Microsoft.RelInfra.Common.Exceptions.ErrorResponseException: Could not find datastore: azureml_managed_openaidevaulttrainingdata..`.

There are no logs or code found at this node.

### Addition information

The notebook code I am referring to is [here](https://github.com/Azure/azureml-examples/blob/main/sdk/python/foundation-models/azure_openai/oai-v2/openai_chat_finetune_pipeline.ipynb).

I'm not sure if the root cause of this error is something in the code/Azure ML workspace. In case it's the former, here's also a list of all the packages and versions in the Jupyter kernel I'm using to run the notebook:
```
- _libgcc_mutex=0.1=main
- _openmp_mutex=5.1=1_gnu
- asttokens=2.2.1=pyhd8ed1ab_0
- backcall=0.2.0=pyh9f0ad1d_0
- backports=1.0=pyhd8ed1ab_3
- backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
- bzip2=1.0.8=h7b6447c_0
- ca-certificates=2023.5.7=hbcca054_0
- debugpy=1.5.1=py310h295c915_0
- decorator=5.1.1=pyhd8ed1ab_0
- entrypoints=0.4=pyhd8ed1ab_0
- executing=1.2.0=pyhd8ed1ab_0
- ipykernel=6.15.0=pyh210e3f2_0
- ipython=8.14.0=pyh41d4057_0
- jedi=0.18.2=pyhd8ed1ab_0
- jupyter_client=7.3.4=pyhd8ed1ab_0
- jupyter_core=5.3.1=py310hff52083_0
- ld_impl_linux-64=2.38=h1181459_1
- libffi=3.4.4=h6a678d5_0
- libgcc-ng=11.2.0=h1234567_1
- libgomp=11.2.0=h1234567_1
- libsodium=1.0.18=h36c2ea0_1
- libstdcxx-ng=11.2.0=h1234567_1
- libuuid=1.41.5=h5eee18b_0
- matplotlib-inline=0.1.6=pyhd8ed1ab_0
- ncurses=6.4=h6a678d5_0
- nest-asyncio=1.5.6=pyhd8ed1ab_0
- openssl=3.0.9=h7f8727e_0
- parso=0.8.3=pyhd8ed1ab_0
- pexpect=4.8.0=pyh1a96a4e_2
- pickleshare=0.7.5=py_1003
- pip=23.1.2=py310h06a4308_0
- platformdirs=3.6.0=pyhd8ed1ab_0
- prompt-toolkit=3.0.38=pyha770c72_0
- prompt_toolkit=3.0.38=hd8ed1ab_0
- psutil=5.9.0=py310h5eee18b_0
- ptyprocess=0.7.0=pyhd3deb0d_0
- pure_eval=0.2.2=pyhd8ed1ab_0
- pygments=2.15.1=pyhd8ed1ab_0
- python=3.10.11=h955ad1f_3
- python-dateutil=2.8.2=pyhd8ed1ab_0
- python_abi=3.10=2_cp310
- pyzmq=25.1.0=py310h6a678d5_0
- readline=8.2=h5eee18b_0
- setuptools=67.8.0=py310h06a4308_0
- six=1.16.0=pyh6c4a22f_0
- sqlite=3.41.2=h5eee18b_0
- stack_data=0.6.2=pyhd8ed1ab_0
- tk=8.6.12=h1ccaba5_0
- tornado=6.1=py310h5764c6d_3
- traitlets=5.9.0=pyhd8ed1ab_0
- typing-extensions=4.6.3=hd8ed1ab_0
- typing_extensions=4.6.3=pyha770c72_0
- wcwidth=0.2.6=pyhd8ed1ab_0
- wheel=0.38.4=py310h06a4308_0
- xz=5.4.2=h5eee18b_0
- zeromq=4.3.4=h9c3ff4c_1
- zlib=1.2.13=h5eee18b_0
- pip:
- adal==1.2.7
- aiosignal==1.3.1
- alembic==1.11.1
- argcomplete==2.1.2
- attrs==23.1.0
- azure-ai-ml==1.12.1
- azure-common==1.1.28
- azure-core==1.27.1
- azure-graphrbac==0.61.1
- azure-identity==1.13.0
- azure-mgmt-authorization==3.0.0
- azure-mgmt-containerregistry==10.1.0
- azure-mgmt-core==1.4.0
- azure-mgmt-keyvault==10.2.2
- azure-mgmt-resource==22.0.0
- azure-mgmt-storage==21.0.0
- azure-storage-blob==12.16.0
- azure-storage-file-datalake==12.11.0
- azure-storage-file-share==12.12.0
- azureml-core==1.51.0.post1
- azureml-dataprep==4.12.1
- azureml-dataprep-native==38.0.0
- azureml-dataprep-rslex==2.19.2
- azureml-fsspec==1.2.0
- azureml-mlflow==1.51.0
- backports-tempfile==1.0
- backports-weakref==1.0.post1
- bcrypt==4.0.1
- blinker==1.6.2
- cachetools==5.3.1
- certifi==2023.5.7
- cffi==1.15.1
- charset-normalizer==3.1.0
- click==8.0.4
- cloudpickle==2.2.1
- colorama==0.4.6
- contextlib2==21.6.0
- contourpy==1.1.0
- cryptography==41.0.1
- cycler==0.11.0
- cython==0.29.35
- databricks-cli==0.17.7
- distlib==0.3.6
- distro==1.8.0
- docker==6.1.3
- dotnetcore2==3.1.23
- filelock==3.12.2
- flask==2.3.2
- fonttools==4.40.0
- frozenlist==1.3.3
- fsspec==2023.6.0
- gitdb==4.0.10
- gitpython==3.1.31
- google-api-core==2.11.1
- google-auth==2.20.0
- googleapis-common-protos==1.59.1
- greenlet==2.0.2
- grpcio==1.43.0
- gunicorn==20.1.0
- humanfriendly==10.0
- idna==3.4
- imageio==2.31.1
- importlib-metadata==6.7.0
- isodate==0.6.1
- itsdangerous==2.1.2
- jeepney==0.8.0
- jinja2==3.1.2
- jmespath==1.0.1
- joblib==1.2.0
- jsonpickle==3.0.1
- jsonschema==4.17.3
- kiwisolver==1.4.4
- knack==0.10.1
- lazy-loader==0.2
- mako==1.2.4
- markdown==3.4.3
- markupsafe==2.1.3
- marshmallow==3.19.0
- matplotlib==3.7.1
- mldesigner==0.1.0b13
- mlflow==2.4.1
- mlflow-skinny==2.4.1
- mltable==1.4.1
- msal==1.22.0
- msal-extensions==1.0.0
- msgpack==1.0.5
- msrest==0.7.1
- msrestazure==0.6.4
- ndg-httpsclient==0.5.1
- networkx==3.1
- numpy==1.25.0
- oauthlib==3.2.2
- opencensus==0.11.2
- opencensus-context==0.1.3
- opencensus-ext-azure==1.1.9
- packaging==23.0
- pandas==2.0.2
- paramiko==3.2.0
- pathspec==0.11.1
- pillow==9.5.0
- pkginfo==1.9.6
- portalocker==2.7.0
- protobuf==3.20.3
- pyarrow==12.0.1
- pyasn1==0.5.0
- pyasn1-modules==0.3.0
- pycparser==2.21
- pydash==7.0.5
- pyjwt==2.7.0
- pynacl==1.5.0
- pyopenssl==23.2.0
- pyparsing==3.1.0
- pyrsistent==0.19.3
- pysocks==1.7.1
- pytz==2023.3
- pywavelets==1.4.1
- pyyaml==6.0
- querystring-parser==1.2.4
- ray==2.0.0
- requests==2.31.0
- requests-oauthlib==1.3.1
- rsa==4.9
- scikit-image==0.21.0
- scikit-learn==1.2.2
- scipy==1.10.1
- secretstorage==3.3.3
- smmap==5.0.0
- sqlalchemy==2.0.16
- sqlparse==0.4.4
- strictyaml==1.7.3
- tabulate==0.9.0
- threadpoolctl==3.1.0
- tifffile==2023.4.12
- tqdm==4.65.0
- tzdata==2023.3
- urllib3==1.26.16
- virtualenv==20.23.1
- websocket-client==1.6.0
- werkzeug==2.3.6
- zipp==3.15.0
```

Contributor guide

Open the contributing guide

Research direction

Start with openai_chat_finetune_pipeline.ipynb and run it in the reported Python 3.10.11 and azure-ai-ml environment against an Azure ML workspace. Inspect the first Data Import node and the reported missing datastore error, then compare the notebook's pipeline and workspace setup. Done means the pipeline job completes its Data Import node and proceeds successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.