Azure / Azure/azure-sdk-for-java

[QUERY] How to use Linked Services with Cosmos DB Spark connector Throughput control in Synapse Analytics

Offen
#42,855 8 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @TheovanKraay Auf GitHub ansehen
Client Cosmos customer-reported needs-team-attention question Service Attention
Vorherrschende Sprache
Java
Sterne
2.6k
Forks
2.2k
Ø Merge
2 T. 7 Std.
Gemergte PRs (30 T.)
211

Beschreibung

**Query/Question**

What is the equivalent to `spark.synapse.linkedService` for use with `spark.cosmos.throughputControl.globalControl.database`

I need to use private endpoints because of DEP, and having credentials stored with Linked Services is very handy.

***Why is this not a Bug or a feature Request?***

Feels like I'm missing something really obvious, but the [docs](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3_2-12/docs/configuration-reference.md) don't seem to cover what I'm seeking,

**Setup (please complete the following information if applicable):**
- OS: Spark 3.4 Python 3.10 Scala
- IDE: Synapse Analytics Web based
- Library/Libraries: default Spark Pool config

**PySpark**
```python
(
df_floodre
.withColumn('id', F.col('uprn').cast('string'))
.write
.format("cosmos.oltp")
.mode("Append")
.option("spark.synapse.linkedService", "CosmosDB_GraphQL_databasets")
.option("spark.cosmos.container", 'floodRe')
.option('spark.cosmos.throughputControl.enabled','true')
.option('spark.cosmos.throughputControl.name','floodRe_Loader_Jobs_Group')
.option('spark.cosmos.throughputControl.globalControl.database','CosmosDB_GraphQL_databasets_ThroughputControl')
.option('spark.cosmos.throughputControl.globalControl.container','ThroughputControl-floodre')
.option('spark.cosmos.throughputControl.targetThroughput',7000)
.save()
)
```

I've tried a few likely nonsensical permutations and got 403 or 404 errors. Guidance much appreciated.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.