apache / apache/auron

JNI Hadoop file wrappers truncate paths containing literal #

Open
#2,257 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

**Describe the bug**
Auron's JNI Hadoop file wrappers may truncate paths containing a literal #.
This happens because `JniBridge` rebuilds paths via `new Path(new URI(path))`, where Java `URI` treats everything after `#` as a fragment, so Hadoop receives a truncated path.

For example, this intended path:
```text
hdfs://mycluster/auron-it-hdfs-rbf-repro/raw#mini.txt
```
is opened as:
```text
/auron-it-hdfs-rbf-repro/raw
```
**To Reproduce**
```python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("auron-hdfs-hash-repro").getOrCreate()
jvm = spark._jvm
conf = spark._jsc.hadoopConfiguration()

path = "hdfs://mycluster/auron-it-hdfs-rbf-repro/raw#mini.txt"
hadoop_path = jvm.org.apache.hadoop.fs.Path(path)
fs = hadoop_path.getFileSystem(conf)

print("path=", path)

if fs.exists(hadoop_path):
fs.delete(hadoop_path, False)

out = fs.create(hadoop_path, True)
out.write(bytearray(b"data"))
out.close()
print("plain_hadoop_create_ok")

plain_in = fs.open(hadoop_path)
plain_in.close()
print("plain_hadoop_open_ok")

jvm.org.apache.auron.jni.JniBridge.openFileAsDataInputWrapper(fs, path).close()
print("jni_open_ok")

spark.stop()
```
The relevant output is:

Image

**Expected behavior**
Auron should preserve Hadoop `Path(String)` semantics when opening or creating files through JNI Hadoop wrappers.
`JniBridge.openFileAsDataInputWrapper` and `JniBridge.createFileAsDataOutputWrapper` should open the exact path string passed by the caller, including literal `#` characters in the Hadoop path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.