issue with ruamel.yaml when trying to install azureml-sdk[databricks] on a Databricks cluster

I have tried installing azureml-sdk[databricks] on a Azure Databricks cluster, runtime 11.0 ML (includes Apache Spark 3.3.0, Scala 2.12), node type Standard_DS_v2 in two different ways
1) Using the UI following the instructions in the Track Azure Databricks experiments in Azure Machine Learning course (https://docs.microsoft.com/en-us/learn/modules/track-azure-databricks-experiments-azure-machine-learning/?ns-enrollment-type=learningpath&ns-enrollment-id=learn.wwl.build-operate-machine-learning-solutions-azure-databricks)
and
2) With !pip install azureml-sdk[databricks]

With (1) installation never completes
With (2) installation fails. The output shows several instances of
INFO: pip is looking at multiple versions of azureml-train to determine which version is compatible with other requirements. This could take a while.
and
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
After about 90 seconds installation fails with the following error message
~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:582:22: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
582 | yaml_char_t *value, int length,
| ~~~~~~~~~~~~~^~~~~
ext/_ruamel_yaml.c:18390:71: warning: pointer targets in passing argument 2 of ‘yaml_sequence_start_event_initialize’ differ in signedness [-Wpointer-sign]
18390 | __pyx_t_3 = ((yaml_sequence_start_event_initialize(__pyx_v_event, __pyx_v_anchor, __pyx_v_tag, __pyx_v_implicit, __pyx_v_sequence_style) == 0) != 0);
| ^~~~~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:604:22: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
604 | yaml_char_t *anchor, yaml_char_t *tag, int implicit,
| ~~~~~~~~~~~~~^~~~~~
ext/_ruamel_yaml.c:18390:87: warning: pointer targets in passing argument 3 of ‘yaml_sequence_start_event_initialize’ differ in signedness [-Wpointer-sign]
18390 | __pyx_t_3 = ((yaml_sequence_start_event_initialize(__pyx_v_event, __pyx_v_anchor, __pyx_v_tag, __pyx_v_implicit, __pyx_v_sequence_style) == 0) != 0);
| ^~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:604:43: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
604 | yaml_char_t *anchor, yaml_char_t *tag, int implicit,
| ~~~~~~~~~~~~~^~~
ext/_ruamel_yaml.c:18824:70: warning: pointer targets in passing argument 2 of ‘yaml_mapping_start_event_initialize’ differ in signedness [-Wpointer-sign]
18824 | __pyx_t_2 = ((yaml_mapping_start_event_initialize(__pyx_v_event, __pyx_v_anchor, __pyx_v_tag, __pyx_v_implicit, __pyx_v_mapping_style) == 0) != 0);
| ^~~~~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:636:22: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
636 | yaml_char_t *anchor, yaml_char_t *tag, int implicit,
| ~~~~~~~~~~~~~^~~~~~
ext/_ruamel_yaml.c:18824:86: warning: pointer targets in passing argument 3 of ‘yaml_mapping_start_event_initialize’ differ in signedness [-Wpointer-sign]
18824 | __pyx_t_2 = ((yaml_mapping_start_event_initialize(__pyx_v_event, __pyx_v_anchor, __pyx_v_tag, __pyx_v_implicit, __pyx_v_mapping_style) == 0) != 0);
| ^~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:636:43: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
636 | yaml_char_t *anchor, yaml_char_t *tag, int implicit,
| ~~~~~~~~~~~~~^~~
ext/_ruamel_yaml.c: In function ‘__pyx_pf_12_ruamel_yaml_8CEmitter_12serialize’:
ext/_ruamel_yaml.c:20303:42: warning: pointer targets in assignment from ‘char *’ to ‘yaml_char_t *’ {aka ‘unsigned char *’} differ in signedness [-Wpointer-sign]
20303 | __pyx_v_tag_directives_end->handle = PyString_AS_STRING(__pyx_v_handle);
| ^
ext/_ruamel_yaml.c:20418:42: warning: pointer targets in assignment from ‘char *’ to ‘yaml_char_t *’ {aka ‘unsigned char *’} differ in signedness [-Wpointer-sign]
20418 | __pyx_v_tag_directives_end->prefix = PyString_AS_STRING(__pyx_v_prefix);
| ^
ext/_ruamel_yaml.c: In function ‘__pyx_f_12_ruamel_yaml_8CEmitter__serialize_node’:
ext/_ruamel_yaml.c:21278:65: warning: pointer targets in passing argument 2 of ‘yaml_alias_event_initialize’ differ in signedness [-Wpointer-sign]
21278 | __pyx_t_2 = ((yaml_alias_event_initialize((&__pyx_v_event), __pyx_v_anchor) == 0) != 0);
| ^~~~~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:555:63: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
555 | yaml_alias_event_initialize(yaml_event_t *event, yaml_char_t *anchor);
| ~~~~~~~~~~~~~^~~~~~
ext/_ruamel_yaml.c:22089:68: warning: pointer targets in passing argument 2 of ‘yaml_scalar_event_initialize’ differ in signedness [-Wpointer-sign]
22089 | __pyx_t_2 = ((yaml_scalar_event_initialize((&__pyx_v_event), __pyx_v_anchor, __pyx_v_tag, __pyx_v_value, __pyx_v_length, __pyx_v_plain_implicit, __pyx_v_quoted_implicit, __pyx_v_scalar_style) == 0) != 0);
| ^~~~~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:581:22: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
581 | yaml_char_t *anchor, yaml_char_t *tag,
| ~~~~~~~~~~~~~^~~~~~
ext/_ruamel_yaml.c:22089:84: warning: pointer targets in passing argument 3 of ‘yaml_scalar_event_initialize’ differ in signedness [-Wpointer-sign]
22089 | __pyx_t_2 = ((yaml_scalar_event_initialize((&__pyx_v_event), __pyx_v_anchor, __pyx_v_tag, __pyx_v_value, __pyx_v_length, __pyx_v_plain_implicit, __pyx_v_quoted_implicit, __pyx_v_scalar_style) == 0) != 0);
| ^~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:581:43: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
581 | yaml_char_t *anchor, yaml_char_t *tag,
| ~~~~~~~~~~~~~^~~
ext/_ruamel_yaml.c:22089:97: warning: pointer targets in passing argument 4 of ‘yaml_scalar_event_initialize’ differ in signedness [-Wpointer-sign]
22089 | __pyx_t_2 = ((yaml_scalar_event_initialize((&__pyx_v_event), __pyx_v_anchor, __pyx_v_tag, __pyx_v_value, __pyx_v_length, __pyx_v_plain_implicit, __pyx_v_quoted_implicit, __pyx_v_scalar_style) == 0) != 0);
| ^~~~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:582:22: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
582 | yaml_char_t *value, int length,
| ~~~~~~~~~~~~~^~~~~
ext/_ruamel_yaml.c:22476:76: warning: pointer targets in passing argument 2 of ‘yaml_sequence_start_event_initialize’ differ in signedness [-Wpointer-sign]
22476 | __pyx_t_2 = ((yaml_sequence_start_event_initialize((&__pyx_v_event), __pyx_v_anchor, __pyx_v_tag, __pyx_v_implicit, __pyx_v_sequence_style) == 0) != 0);
| ^~~~~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:604:22: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
604 | yaml_char_t *anchor, yaml_char_t *tag, int implicit,
| ~~~~~~~~~~~~~^~~~~~
ext/_ruamel_yaml.c:22476:92: warning: pointer targets in passing argument 3 of ‘yaml_sequence_start_event_initialize’ differ in signedness [-Wpointer-sign]
22476 | __pyx_t_2 = ((yaml_sequence_start_event_initialize((&__pyx_v_event), __pyx_v_anchor, __pyx_v_tag, __pyx_v_implicit, __pyx_v_sequence_style) == 0) != 0);
| ^~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:604:43: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
604 | yaml_char_t *anchor, yaml_char_t *tag, int implicit,
| ~~~~~~~~~~~~~^~~
ext/_ruamel_yaml.c:23006:75: warning: pointer targets in passing argument 2 of ‘yaml_mapping_start_event_initialize’ differ in signedness [-Wpointer-sign]
23006 | __pyx_t_2 = ((yaml_mapping_start_event_initialize((&__pyx_v_event), __pyx_v_anchor, __pyx_v_tag, __pyx_v_implicit, __pyx_v_mapping_style) == 0) != 0);
| ^~~~~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:636:22: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
636 | yaml_char_t *anchor, yaml_char_t *tag, int implicit,
| ~~~~~~~~~~~~~^~~~~~
ext/_ruamel_yaml.c:23006:91: warning: pointer targets in passing argument 3 of ‘yaml_mapping_start_event_initialize’ differ in signedness [-Wpointer-sign]
23006 | __pyx_t_2 = ((yaml_mapping_start_event_initialize((&__pyx_v_event), __pyx_v_anchor, __pyx_v_tag, __pyx_v_implicit, __pyx_v_mapping_style) == 0) != 0);
| ^~~~~~~~~~~
| |
| char *
In file included from ext/_ruamel_yaml.h:2,
from ext/_ruamel_yaml.c:523:
ext/yaml.h:636:43: note: expected ‘yaml_char_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
636 | yaml_char_t *anchor, yaml_char_t *tag, int implicit,
| ~~~~~~~~~~~~~^~~
ext/_ruamel_yaml.c: In function ‘PyInit__ruamel_yaml’:
ext/_ruamel_yaml.c:25793:33: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
25793 | __pyx_type_12_ruamel_yaml_Mark.tp_print = 0;
| ^
ext/_ruamel_yaml.c:25810:36: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
25810 | __pyx_type_12_ruamel_yaml_CParser.tp_print = 0;
| ^
ext/_ruamel_yaml.c:25821:37: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
25821 | __pyx_type_12_ruamel_yaml_CEmitter.tp_print = 0;
| ^
ext/_ruamel_yaml.c: In function ‘__Pyx_ParseOptionalKeywords’:
ext/_ruamel_yaml.c:26908:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
26908 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
ext/_ruamel_yaml.c:26908:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
26908 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
| ^~~~~~~~~~~~~~~~~~~
ext/_ruamel_yaml.c:26908:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
26908 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
ext/_ruamel_yaml.c:26908:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
26908 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
ext/_ruamel_yaml.c:26908:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
26908 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
| ^~~~~~~~~~~~~~~~~~~
ext/_ruamel_yaml.c:26908:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
26908 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
ext/_ruamel_yaml.c:26924:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
26924 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
ext/_ruamel_yaml.c:26924:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
26924 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
| ^~~~~~~~~~~~~~~~~~~
ext/_ruamel_yaml.c:26924:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
26924 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
ext/_ruamel_yaml.c:26924:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
26924 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
ext/_ruamel_yaml.c:26924:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
26924 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
| ^~~~~~~~~~~~~~~~~~~
ext/_ruamel_yaml.c:26924:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
26924 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
| ^
In file included from /usr/include/python3.9/unicodeobject.h:1026,
from /usr/include/python3.9/Python.h:97,
from ext/_ruamel_yaml.c:4:
/usr/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> ruamel.yaml

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

This happened after upgrading pip from 21.2.4 to 22.1.2. With pip version 22.1.2 installation failed after more than 6 hours and the error message was

ERROR: Exception:
Traceback (most recent call last):
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-443996c2-41a0-4a83-813c-eea1c5b1d372/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 173, in _main
status = self.run(options, args)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-443996c2-41a0-4a83-813c-eea1c5b1d372/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 203, in wrapper
return func(self, options, args)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-443996c2-41a0-4a83-813c-eea1c5b1d372/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 315, in run
requirement_set = resolver.resolve(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-443996c2-41a0-4a83-813c-eea1c5b1d372/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-443996c2-41a0-4a83-813c-eea1c5b1d372/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 472, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-443996c2-41a0-4a83-813c-eea1c5b1d372/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 383, in resolve
raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2000000

Is this a know problem with a solution? Is it necessary to specify a version of the sdk for installation to succeed?