[FEATURE REQUEST]: support for something like dbutils to move/copy/delete data easily
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
**Is your feature request related to a problem? Please describe.**
There is a request to save one single file in ADLS Gen2 as the output rather than a folder containing several files with partition GUID. In order to do it, in scala, we can play a trick by using dutils: coalesce the files into one and save the folder, copy the merged csv file and rename it, remove the temp folder.
A sample code is like below,
val fileprefix= "abfss://filesystem@storageaccount.dfs.core.windows.net/PageView/V1"
agg_pageView_df.coalesce(1)
.write
.format("com.databricks.spark.csv")
.mode("overwrite")
.option("header", "true")
.option("inferSchema", "true")
.save(fileprefix + ".tmp")
val partition_path = dbutils.fs.ls(fileprefix+".tmp/")
.filter(file=>file.name.endsWith(".csv"))(0).path
dbutils.fs.cp(partition_path,fileprefix+".csv")
dbutils.fs.rm(fileprefix+".tmp",recurse=true)
There is no support like dbutils in Spark.Net. We may need to use a separate ADLS client to connect to the storage and move/delete/copy the data.
That will be great if Spark.Net support this kind of data processing.
**Describe the solution you'd like**
Add support for something like dbutils, then you don't need to connect to the storage again with an independent client (like ALDS client) as spark already connect to it.
Contributor guide
Assessment
This issue has not been assessed yet.