Azure / Azure/azure-data-lake-store-python
Performance issue of azure.datalake.store.core.AzureDLFile.write() for Cosmos
- Lingua principale
- Python
- Stelle
- 76
- Fork
- 79
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
### Description
I ran benchmark to compare the performance to upload data to Cosmos between **AzureDLFile.write()** and the write API from **native libhdfs.so**. The result show there is significant gap. To write same amount data to Cosmos, the time used by azure data lake store is **more than double** of the time HDFS used. I also checked the network throughput, with HDFS we can push it to about **4Gb.** And for ADL the throughput is only reach to **1.3Gb.**
In my testing, I used multiple thread to write the data. Each thread creates individual file and write data into it. I tried to increase the thread number and the buffer size. it didn't help to improve the performance.
My questions are:
1. Is this performance gap expected? Since azure data lake store is based on REST API.
2. Is there any advanced API or parameter I can try to improve the throughput? For my scenario, we have to use the streaming write API to upload the data.
### Environment summary
**SDK Version:** What version of the SDK are you using? (pip show azure-datalake-store)
Answer here: The latest.
**Python Version:** What Python version are you using? Is it 64-bit or 32-bit?
Answer here: python version: 3.6.9 64
**OS Version:** What OS and version are you using?
Answer here: Ubuntu 18.04
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia da azure.datalake.store.core.AzureDLFile.write() e dal confronto riportato con libhdfs.so nativa. Riproduci il benchmark di scrittura in streaming su Python 3.6.9 e Ubuntu 18.04, quindi determina se il divario di throughput è previsto oppure se una API o un parametro documentato lo migliora.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- azure, python
- Ambito
- cloud, data-engineering
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 25/100