apache / apache/beam

Unicode chars in local file system paths break dataflow runner tests.

Open
#18,213 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

toResourceIdentifier("/根目录/") failed with following error, and I think it fails with any unicode chars, probably related to:
http://stackoverflow.com/questions/37409379/invalidpathexception-for-chinese-filename

testGetCurrentDirectoryInUnix(org.apache.beam.sdk.io.LocalResourceIdTest) Time elapsed: 0.011 sec <<< ERROR!
java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: /???
at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
at sun.nio.fs.UnixPath.(UnixPath.java:71)
at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
at java.nio.file.Paths.get(Paths.java:84)
at org.apache.beam.sdk.io.LocalResourceIdTest.toResourceIdentifier(LocalResourceIdTest.java:244)
at org.apache.beam.sdk.io.LocalResourceIdTest.testGetCurrentDirectoryInUnix(LocalResourceIdTest.java:204)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)

Imported from Jira [BEAM-1453](https://issues.apache.org/jira/browse/BEAM-1453). Original Jira may contain additional context.
Reported by: peihe0@gmail.com.

Contributor guide

Open the contributing guide

Research direction

Start with org.apache.beam.sdk.io.LocalResourceIdTest, especially toResourceIdentifier and testGetCurrentDirectoryInUnix, and run the failing test with the reported Unicode path. Trace the path handling used there and verify the test completes without InvalidPathException for Unicode characters.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
operating-systems, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.