CERTIFICATE_VERIFY_FAILED while connecting to the databricks sql endpoint using python through M2M auth

Start with main.py, the Config initialization in credential_provider(), and the traceback request to /oidc/.well-known/oauth-authorization-server. Reproduce the failure with the listed package versions and inspect how the Python SDK handles certificate verification during OAuth M2M setup; done means identifying a supported resolution or documenting the limitation for this environment.

main.py content

from dotenv import load_dotenv, dotenv_values
load_dotenv()
from databricks.sdk.core import Config, oauth_service_principal
from databricks import sql
import os

server_hostname = os.getenv("DATABRICKS_HOST")

def credential_provider():
  config = Config(
    host          = f"https://{server_hostname}",
    client_id     = os.getenv("DATABRICKS_CLIENT_ID"),
    client_secret = os.getenv("DATABRICKS_CLIENT_SECRET"))
  print(config)
  return oauth_service_principal(config)

with sql.connect(server_hostname      = server_hostname,
                 http_path            = f'{os.getenv("DATABRICKS_HTTP_PATH")}',
                 credentials_provider = credential_provider,
                _tls_no_verify=True,
                 ) as connection:
    with connection.cursor() as cursor:
        cursor.execute("SELECT 1 as a")
        print(cursor.fetchall())
        cursor.close()

connection.close()

.env content

DATABRICKS_HOST="adb-XXXXXXXXXXXXX.azuredatabricks.net"

DATABRICKS_CLIENT_ID="XXXXXXX"

DATABRICKS_CLIENT_SECRET="XXXXXXX"

DATABRICKS_HTTP_PATH="/sql/1.0/warehouses/XXXXXXXXXX"

ERROR

Traceback (most recent call last):
  File "/Users/saurabhkumar/PycharmProjects/pythonProject/test.py", line 28, in <module>
    credential_provider()
  File "/Users/saurabhkumar/PycharmProjects/pythonProject/test.py", line 21, in credential_provider
    config = Config(
  File "/Users/saurabhkumar/PycharmProjects/pythonProject/.venv/lib/python3.8/site-packages/databricks/sdk/config.py", line 127, in __init__
    raise ValueError(message) from e
ValueError: default auth: oauth-m2m: HTTPSConnectionPool(host='adb-XXXXXXXXXXXX.azuredatabricks.net', port=443): Max retries exceeded with url: /oidc/.well-known/oauth-authorization-server (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)'))). Config: host=https://adb-XXXXXXXXXXX.azuredatabricks.net, client_id=XXXXXXXXXXX, client_secret=***. Env: DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET

Packages

cachetools==5.5.0
certifi==2024.8.30
charset-normalizer==3.3.2
databricks==0.2
databricks-connect==13.0.1
databricks-sdk==0.32.1
databricks-sql-connector==3.4.0
et-xmlfile==1.1.0
google-auth==2.34.0
googleapis-common-protos==1.65.0
grpcio==1.66.1
grpcio-status==1.66.1
idna==3.8
lz4==4.3.3
numpy==1.24.4
oauthlib==3.2.2
openpyxl==3.1.5
pandas==1.5.3
pip-system-certs==4.0
protobuf==5.28.1
py4j==0.10.9.7
pyarrow==16.1.0
pyasn1==0.6.1
pyasn1_modules==0.4.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
pytz==2024.2
requests==2.32.3
rsa==4.9
six==1.16.0
thrift==0.20.0
urllib3==2.2.2
wrapt==1.16.0