apache / apache/beam

TextIO.Write not handling URI properly

Open
#18,696 0 comments 0 reactions 0 assignees View on GitHub
bug core java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

I think I have found a bug in TextIO, in the way it handles URIs. I am told that the TextIO.Write.to(String) method should take a URI string, but this doesn't seem to work for me.

Test case inline:

```

import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;

import
java.io.File;
import java.io.IOException;
import java.net.URI;
import java.nio.file.*;
import java.util.*;
import
org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
import
org.apache.beam.sdk.values.PCollection;
import org.junit.Rule;
import org.junit.Test;

public class
Beam3429Test {

@Rule
public final transient TestPipeline pipeline = TestPipeline.create();

@Test
public void testBeamTextIO() throws IOException {
List words = Arrays.asList("tom",
"huck", "polly");
Create.Values source = Create.of(words);
PCollection
coll = this.pipeline.apply(source);
String fileName = "test" + System.currentTimeMillis() +
".txt";
String fileURI = new File(fileName).toPath().toUri().toString();
// ie file:///full/unix/path/to/test.file

coll.apply(TextIO.write().to(fileURI));
//test passes if we use the line below instead
of the above
// coll.apply(TextIO.write().to(fileName));
pipeline.run();


//read all sharded files:
Path dir = Paths.get(URI.create(fileURI)).getParent();


List files = new ArrayList<>();
Files.newDirectoryStream(dir, new DirectoryStream.Filter()
{
@Override public boolean accept(Path entry) throws IOException {
return
entry.toString().contains(fileName);
}
}).forEach(path -> files.add(path));

assertFalse("no files produced!", files.isEmpty());
List fileContents = new
ArrayList<>();
for (Path f : files) {
fileContents.addAll(Files.readAllLines(f));

}
assertTrue(fileContents.contains("tom"));
assertTrue(fileContents.contains("polly"));

}
}

```

Imported from Jira [BEAM-3429](https://issues.apache.org/jira/browse/BEAM-3429). Original Jira may contain additional context.
Reported by: mdarwin.

Contributor guide

Open the contributing guide

Research direction

Start with the inline Beam3429Test reproduction and the TextIO.write().to(String) entry point. Run the test with a file URI and compare it with the plain-path case. Done means URI output creates the expected sharded files and the existing content assertions pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.