Azure Databricks MDS write ops in error: MapInPandas write_mds gives message Spark higher-order functions are not supported in Unity Catalog

Environment

Azure Databricks 14.3 LTS used on a distributed cluster.

To reproduce

Steps to reproduce the behavior:

1. Code as follow:

d = {
    'code': ['test', 'test', 'test'], 
    'age': ['test', 'test', 'test'], 
    'id': ['ib4', '7h!', '67h']
}

df_test = ps.DataFrame(data=d).to_spark()

mds_kwargs = {
    'out': 'test', 
    'columns': {
        'code': 'str', 
        'age': 'str',
        'id': 'str',
    },
    'keep_local': True
}

dataframe_to_mds(
    dataframe=df_test,
    merge_index=True,
    mds_kwargs=mds_kwargs
)

Expected behavior

The script should result in a write op that writes the created DataFrame on the Unity Catalog / local persistence in the MDS format.

Additional context

Documentation is not accessible today, we are using version 0.7.x .

Start by reproducing the distributed-cluster example through dataframe_to_mds, focusing on the MapInPandas/write_mds path and the Unity Catalog error. Confirm that the DataFrame can be written to Unity Catalog or local persistence in MDS format without the reported higher-order-functions message.