Azure / Azure/azure-sdk-for-python

mltable produces key error when trying to consume sdk v1 dataset type data with provided microsoft consume code

Open
#38,944 4 comments 0 reactions 1 assignee Claimed by @vivram View on GitHub
Client customer-reported dataprep Machine Learning needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

- **Package Name**: mltable
- **Package Version**: 1.6.1
- **Operating System**: Ubuntu 20.04
- **Python Version**: 3.10.14

**Describe the bug**
While trying to consume a data asset from azure machine learning studio with table type but with underlying dataset type tabular (see attached image under screenshots) using the microsoft supplied example code for reading this asset into a pandas dataframe via an mltable object, we encounter a `KeyError` with key `paths` missing as shown below in the error trace.

Full error trace

```python
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In[3], line 8
5 ml_client = MLClient.from_config(credential=DefaultAzureCredential())
6 data_asset = ml_client.data.get("Energie_Aansluitingen_Current_1000", version="1")
----> 8 tbl = mltable.load(f'azureml:/{data_asset.id}')
10 df = tbl.to_pandas_dataframe()
11 df

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azureml/dataprep/api/_loggerfactory.py:279, in track..monitor..wrapper(*args, **kwargs)
277 with _LoggerFactory.track_activity(logger, func.__name__, activity_type, custom_dimensions) as activityLogger:
278 try:
--> 279 return func(*args, **kwargs)
280 except Exception as e:
281 if hasattr(activityLogger, ACTIVITY_INFO_KEY) and hasattr(e, ERROR_CODE_KEY):

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/mltable/mltable.py:600, in load(uri, storage_options, ml_client)
547 @track(_get_logger,activity_type=_PUBLIC_API, custom_dimensions={'app_name': _APP_NAME})
548 def load(uri, storage_options: dict = None, ml_client= None):
549 """
550 Loads the MLTable file (YAML) present at the given uri.
551
(...)
598 :rtype: mltable.MLTable
599 """
--> 600 return _load(uri, storage_options, True, ml_client)

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azureml/dataprep/api/_loggerfactory.py:279, in track..monitor..wrapper(*args, **kwargs)
277 with _LoggerFactory.track_activity(logger, func.__name__, activity_type, custom_dimensions) as activityLogger:
278 try:
--> 279 return func(*args, **kwargs)
280 except Exception as e:
281 if hasattr(activityLogger, ACTIVITY_INFO_KEY) and hasattr(e, ERROR_CODE_KEY):

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/mltable/mltable.py:706, in _load(uri, storage_options, enable_validate, ml_client)
704 return mltable_loaded
705 except Exception as ex:
--> 706 _reclassify_rslex_error(ex)

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azureml/dataprep/api/mltable/_validation_and_error_handler.py:90, in _reclassify_rslex_error(err)
87 if 'ExecutionError(StreamError(PermissionDenied' in err_msg:
88 raise UserErrorException(
89 f'Getting permission error please make sure proper access is configured on storage: {err_msg}')
---> 90 raise err

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/mltable/mltable.py:698, in _load(uri, storage_options, enable_validate, ml_client)
696 # v1 sql dataset doesnt have paths
697 if og_path_pairs is None: # may have been set in _load_mltable_from_data_asset_uri
--> 698 mltable_dict, og_path_pairs = _make_all_paths_absolute(mltable_dict, base_path)
699 mltable_loaded = MLTable._create_from_dict(mltable_yaml_dict=mltable_dict,
700 path_pairs=og_path_pairs,
701 load_uri=load_uri)
702 mltable_loaded._workspace_context = _parse_workspace_context_from_longform_uri(load_uri)

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/mltable/_utils.py:74, in _make_all_paths_absolute(mltable_yaml_dict, base_path)
72 mltable_yaml_dict[_PATHS_KEY] = list(map(lambda x: x[1], path_pairs))
73 else:
---> 74 path_pairs = list(tuple(zip(mltable_yaml_dict[_PATHS_KEY], mltable_yaml_dict[_PATHS_KEY])))
75 return mltable_yaml_dict, path_pairs

KeyError: 'paths'
```

**To Reproduce**
1. Setup an Azure SQL database type datastore in azure ml studio.
2. Create a data asset from the datastore using a sql statement and make sure it can connect and has data.
3. Try to consume the data asset for interactive development using the supplied microsoft snippet in the data asset section on azure ml (see second screenshot)

**Expected behavior**
I expected to end up with a pandas dataframe.

**Screenshots**

Screenshot of the data asset in azure ml

![Image](https://github.com/user-attachments/assets/1c563a1b-a765-4ad5-81ed-67b6224faee7)

Screenshot of the consume code snippet

![Image](https://github.com/user-attachments/assets/11891d84-b5ed-4db5-bd16-e9b916a734c0)

Screenshot of the error trace

![Image](https://github.com/user-attachments/assets/f6858b9d-1756-4fc7-a0c5-676fd0e2e589)

**Additional context**
We run this code on a Standard_DS12_v2 (4 cores, 28 GB RAM, 56 GB disk) compute instance with:
```
azure-ai-ml==1.23.0
azure-identity==1.18.0
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.