Azure / Azure/azure-sdk-for-java

[QUERY] How to use Linked Services with Cosmos DB Spark connector Throughput control in Synapse Analytics

Open
#42,855 8 comments 0 reactions 1 assignee Claimed by @TheovanKraay View on GitHub
Client Cosmos customer-reported needs-team-attention question Service Attention
Dominant language
Java
Stars
2.6k
Forks
2.2k
Avg merge
2d 8h
Merged PRs (30d)
178

Description

**Query/Question**

What is the equivalent to `spark.synapse.linkedService` for use with `spark.cosmos.throughputControl.globalControl.database`

I need to use private endpoints because of DEP, and having credentials stored with Linked Services is very handy.

***Why is this not a Bug or a feature Request?***

Feels like I'm missing something really obvious, but the [docs](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3_2-12/docs/configuration-reference.md) don't seem to cover what I'm seeking,

**Setup (please complete the following information if applicable):**
- OS: Spark 3.4 Python 3.10 Scala
- IDE: Synapse Analytics Web based
- Library/Libraries: default Spark Pool config

**PySpark**
```python
(
df_floodre
.withColumn('id', F.col('uprn').cast('string'))
.write
.format("cosmos.oltp")
.mode("Append")
.option("spark.synapse.linkedService", "CosmosDB_GraphQL_databasets")
.option("spark.cosmos.container", 'floodRe')
.option('spark.cosmos.throughputControl.enabled','true')
.option('spark.cosmos.throughputControl.name','floodRe_Loader_Jobs_Group')
.option('spark.cosmos.throughputControl.globalControl.database','CosmosDB_GraphQL_databasets_ThroughputControl')
.option('spark.cosmos.throughputControl.globalControl.container','ThroughputControl-floodre')
.option('spark.cosmos.throughputControl.targetThroughput',7000)
.save()
)
```

I've tried a few likely nonsensical permutations and got 403 or 404 errors. Guidance much appreciated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.