Azure / Azure/azureml-examples

Data Asset creation for randomly spitted Mltables gives error in AzureML SDK v2

Open
#3,223 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Operating System

Windows

### Version Information
Python 3.10

attrs 23.2.0
azure-ai-ml 1.16.1
azure-common 1.1.28
azure-core 1.30.1
azure-identity 1.16.0
azure-mgmt-core 1.4.0
azure-storage-blob 12.20.0
azure-storage-file-datalake 12.15.0
azure-storage-file-share 12.16.0
azureml-dataprep 5.1.6
azureml-dataprep-native 41.0.0
azureml-dataprep-rslex 2.22.2
cachetools 5.3.3
certifi 2024.6.2
cffi 1.16.0
charset-normalizer 3.3.2
cloudpickle 2.2.1
colorama 0.4.6
cryptography 42.0.8
google-api-core 2.19.0
google-auth 2.29.0
googleapis-common-protos 1.63.1
idna 3.7
isodate 0.6.1
jsonschema 4.22.0
jsonschema-specifications 2023.12.1
marshmallow 3.21.3
mltable 1.6.1
msal 1.28.0
msal-extensions 1.1.0
msrest 0.7.1
numpy 1.26.4
oauthlib 3.2.2
opencensus 0.11.4
opencensus-context 0.1.3
opencensus-ext-azure 1.1.13
opencensus-ext-logging 0.1.1
packaging 24.0
pandas 2.2.2
pip 24.0
portalocker 2.8.2
proto-plus 1.23.0
protobuf 4.25.3
psutil 5.9.8
pyarrow 16.1.0
pyasn1 0.6.0
pyasn1_modules 0.4.0
pycparser 2.22
pydash 8.0.1
PyJWT 2.8.0
python-dateutil 2.9.0.post0
pytz 2024.1
pywin32 306
PyYAML 6.0.1
referencing 0.35.1
requests 2.32.3
requests-oauthlib 2.0.0
rpds-py 0.18.1
rsa 4.9
setuptools 69.5.1
six 1.16.0
strictyaml 1.7.3
tqdm 4.66.4
typing_extensions 4.12.1
tzdata 2024.1
urllib3 2.2.1
wheel 0.43.0

### Steps to reproduce

import time
import mltable
from azure.ai.ml import MLClient
from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes
from azure.identity import DefaultAzureCredential

subscription_id = "<>"
resource_group = "<>"
workspace = "<>"

VERSION = time.strftime("%Y.%m.%d.%H%M%S", time.gmtime())

ml_client = MLClient(
DefaultAzureCredential(), subscription_id, resource_group, workspace
)

data_asset = ml_client.data.get("test-sdkv2", version="2024.06.06.105241")
paths = [{ "folder" : data_asset.path }]

data_mltable = mltable.from_delimited_files(paths, header='all_files_same_headers', delimiter=',' , encoding='utf8', empty_as_string=False)
train_table, test_table = data_mltable.random_split(0.8, seed=223)

train_table.save("./train")
test_table.save("./test")

train_data_asset = Data(
path="./train",
type=AssetTypes.MLTABLE,
description="Training Dataset",
name="SDK2TrainingDataset",
version=VERSION,
)

test_data_asset = Data(
path="./test",
type=AssetTypes.MLTABLE,
description="Testing Dataset",
name="SDK2TestingDataset",
version=VERSION,
)

ml_client.data.create_or_update(train_data_asset )
ml_client.data.create_or_update(test_data_asset )

### Expected behavior

Two Data Assets should be created:

1) SDK2TrainingDataset
2) SDK2TestingDataset

### Actual behavior

Traceback (most recent call last):
File "D:\AMLSDK2\run.py", line 77, in
ml_client.data.create_or_update(train_data_asset)
File "C:\Users\sshaharyaar\AppData\Local\miniconda3\envs\conda_azureml_sdkv2\lib\site-packages\azure\ai\ml\_telemetry\activity.py", line 292, in wrapper
return f(*args, **kwargs)
File "C:\Users\sshaharyaar\AppData\Local\miniconda3\envs\conda_azureml_sdkv2\lib\site-packages\azure\ai\ml\operations\_data_operations.py", line 425, in create_or_update
raise ex
File "C:\Users\sshaharyaar\AppData\Local\miniconda3\envs\conda_azureml_sdkv2\lib\site-packages\azure\ai\ml\operations\_data_operations.py", line 363, in create_or_update
referenced_uris = self._validate(data)
File "C:\Users\sshaharyaar\AppData\Local\miniconda3\envs\conda_azureml_sdkv2\lib\site-packages\azure\ai\ml\_telemetry\activity.py", line 292, in wrapper
return f(*args, **kwargs)
File "C:\Users\sshaharyaar\AppData\Local\miniconda3\envs\conda_azureml_sdkv2\lib\site-packages\azure\ai\ml\operations\_data_operations.py", line 559, in _validate
validate_mltable_metadata(
File "C:\Users\sshaharyaar\AppData\Local\miniconda3\envs\conda_azureml_sdkv2\lib\site-packages\azure\ai\ml\_utils\_data_utils.py", line 68, in validate_mltable_metadata
err_path = ".".join(error.path)
TypeError: sequence item 1: expected str instance, int found

### Addition information

I am using the Credit Card Clients dataset which is available in Azure SDK v2 Documentation. Moreover, I found no example in SDK v2 documentation where data is being split in train and test and then uploaded as data assets.

Contributor guide

Open the contributing guide

Research direction

Reproduce the provided Python script with azure-ai-ml 1.16.1 and inspect validate_mltable_metadata in azure/ai/ml/_utils/_data_utils.py, identified in the traceback. Done means the train and test MLTABLE directories can both be submitted as Data assets without the TypeError, with a regression test or a narrowed failure case.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.