dotnet / dotnet/spark

[FEATURE REQUEST]: support for something like dbutils to move/copy/delete data easily

Open
#328 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

**Is your feature request related to a problem? Please describe.**
There is a request to save one single file in ADLS Gen2 as the output rather than a folder containing several files with partition GUID. In order to do it, in scala, we can play a trick by using dutils: coalesce the files into one and save the folder, copy the merged csv file and rename it, remove the temp folder.
A sample code is like below,

val fileprefix= "abfss://filesystem@storageaccount.dfs.core.windows.net/PageView/V1"

agg_pageView_df.coalesce(1)
.write
.format("com.databricks.spark.csv")
.mode("overwrite")
.option("header", "true")
.option("inferSchema", "true")
.save(fileprefix + ".tmp")

val partition_path = dbutils.fs.ls(fileprefix+".tmp/")
.filter(file=>file.name.endsWith(".csv"))(0).path

dbutils.fs.cp(partition_path,fileprefix+".csv")
dbutils.fs.rm(fileprefix+".tmp",recurse=true)

There is no support like dbutils in Spark.Net. We may need to use a separate ADLS client to connect to the storage and move/delete/copy the data.

That will be great if Spark.Net support this kind of data processing.

**Describe the solution you'd like**
Add support for something like dbutils, then you don't need to connect to the storage again with an independent client (like ALDS client) as spark already connect to it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.