JNI Hadoop file wrappers truncate paths containing literal #
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 21
Description
**Describe the bug**
Auron's JNI Hadoop file wrappers may truncate paths containing a literal #.
This happens because `JniBridge` rebuilds paths via `new Path(new URI(path))`, where Java `URI` treats everything after `#` as a fragment, so Hadoop receives a truncated path.
For example, this intended path:
```text
hdfs://mycluster/auron-it-hdfs-rbf-repro/raw#mini.txt
```
is opened as:
```text
/auron-it-hdfs-rbf-repro/raw
```
**To Reproduce**
```python
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("auron-hdfs-hash-repro").getOrCreate()
jvm = spark._jvm
conf = spark._jsc.hadoopConfiguration()
path = "hdfs://mycluster/auron-it-hdfs-rbf-repro/raw#mini.txt"
hadoop_path = jvm.org.apache.hadoop.fs.Path(path)
fs = hadoop_path.getFileSystem(conf)
print("path=", path)
if fs.exists(hadoop_path):
fs.delete(hadoop_path, False)
out = fs.create(hadoop_path, True)
out.write(bytearray(b"data"))
out.close()
print("plain_hadoop_create_ok")
plain_in = fs.open(hadoop_path)
plain_in.close()
print("plain_hadoop_open_ok")
jvm.org.apache.auron.jni.JniBridge.openFileAsDataInputWrapper(fs, path).close()
print("jni_open_ok")
spark.stop()
```
The relevant output is:
**Expected behavior**
Auron should preserve Hadoop `Path(String)` semantics when opening or creating files through JNI Hadoop wrappers.
`JniBridge.openFileAsDataInputWrapper` and `JniBridge.createFileAsDataOutputWrapper` should open the exact path string passed by the caller, including literal `#` characters in the Hadoop path.
Contributor guide
Assessment
This issue has not been assessed yet.