dotnet / dotnet/spark

[BUG]: "Exists" does not work on Synapse

Open
#970 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

**Describe the bug**
We are trying to use the new Exists feature in .NET for Apache Spark 2.0 on Azure Synapse.
Unfortunately, the corresponding function returns "False" for every path, regardless of whether it points to a valid file or not.

**To Reproduce**
In our tests, we have the file directly in Synapse primary data lake.

The file can be read without any problems using:

spark.Read().Schema(schema).Parquet(sourcepath)

We use the following code to check if the file exists or not:

FileSystem fs = FileSystem.Get(spark.SparkContext.HadoopConfiguration());
bool fileExists = fs.Exists(sourcepath);

We are using Apache Spark Version 3.1 with .NET for Apache Spark 2.0.0.

**Expected behavior**
If the path points to an existing file, the function should return "True".

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.