Sentinel Data Lake Query Provider Support

Is your feature request related to a problem? Please describe.

We have a subset of data stored as lake-only-mode tables in sentinel which avoids cost, as well as for archive, but I cannot find a documented query provider I can use for this with msticpy.

Describe the solution you'd like

I'd like a data driver/query provider implementation that supports sentinel data lake features.

Describe alternatives you've considered

As a workaround, an external table definition can be created via ADX. See: https://learn.microsoft.com/en-us/azure/sentinel/datalake/kql-queries#query-the-data-lake-with-azure-data-explorer

However, the way the existing ADX data driver / query provider wraps authentication does not work well and below is an example of vibe coding past the exceptions with monkey patches.

# FIXME: monkey patch the missmatch auth_types vs auth_methods as the Kusto driver calls az_connect with auth_types instead of auth_methods which is what the az_connect function expects?
import msticpy.data.drivers.azure_kusto_driver as akd
from msticpy.auth.azure_auth import az_connect as _orig_az_connect
def _az_connect_compat(*, auth_types=None, auth_methods=None, **kwargs):
    if auth_methods is None and auth_types is not None:
        auth_methods = auth_types
    return _orig_az_connect(auth_methods=auth_methods, **kwargs)

akd.az_connect = _az_connect_compat


# FIXME: monkey patch so that correct KustoServiceResourceId is obtained from metadata endpoint. Assuming <https://api.securityplatform.microsoft.com/.default> does not work and results in AADSTS500011.
import requests
def _scope_from_cluster_uri(resource_uri: str) -> str:
    try:
        meta = requests.get(
            resource_uri.rstrip("/") + "/v1/rest/auth/metadata",
            timeout=30
        ).json()
        resource_id = meta["AzureAD"]["KustoServiceResourceId"]
        return f"{resource_id}/.default"
    except Exception:
        # safe fallback for regular ADX clusters
        return "https://kusto.kusto.windows.net/.default"

akd.get_default_resource_name = _scope_from_cluster_uri


sentinel_data_lake_adx_endpoint = 'https://api.securityplatform.microsoft.com/lake/kql'
database = '<workspace name>-<workspace id>'
q_prov_data_lake_via_adx = mp.QueryProvider('Kusto')
q_prov_data_lake_via_adx.connect(
    cluster=sentinel_data_lake_adx_endpoint, database=database,
    # FIXME: msticpy chained credentials and vscode auth did not work so use other auth types for now.
    auth_types=["cli", "interactive"]
)

Additional context

As per https://learn.microsoft.com/en-us/azure/sentinel/datalake/notebooks-overview, there's a lot more to Sentinel Data Lake besides running KQL queries and a more "native driver" implementation instead of wrapping ADX could expose more features, e.g. pyspark.

Start by reviewing the existing Azure Kusto data driver/query provider and its authentication integration, then compare them with Sentinel Data Lake KQL query and notebook documentation. Define the supported provider scope, including whether native features beyond KQL are included, and verify that documented authentication and querying work without the reported monkey patches.