apache / apache/hugegraph-toolchain

Merge LocalFileReader and HDFSFileReader

Open
#115 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
121
Forks
125
Avg merge
10d 10h
Merged PRs (30d)
3

Description

The code is as follows:
```
public class HadoopAndLocalReadTest {

private static final long BUF_SIZE = 4 * 1024 *1024;

@Test
public void testHadoopReadLocal() throws IOException {

long startTime=System.nanoTime();

Path file = new Path("file:///Downloads/index-00001");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);

FSDataInputStream getIt = fs.open(file);
BufferedReader d = new BufferedReader(new InputStreamReader(getIt),
(int) BUF_SIZE);
String lineTxt = null;
while ((lineTxt = d.readLine()) != null) {
}
long endTime=System.nanoTime();
System.out.println("The elapsed time of Hadoop: " + (endTime - startTime) + "ns");
d.close();
fs.close();

}

@Test
public void testReadLocal() throws IOException {
long startTime=System.nanoTime();
String filePath = "/Downloads/index-00001";
File file = new File(filePath);
InputStreamReader read = new InputStreamReader(
new FileInputStream(file));
BufferedReader bufferedReader = new BufferedReader(read, (int) BUF_SIZE);

String lineTxt = null;
while ((lineTxt = bufferedReader.readLine()) != null) {
}
long endTime=System.nanoTime();
System.out.println("The elapsed time of local : " + (endTime - startTime) + "ns");
bufferedReader.close();
read.close();

}
}
```

The number of test sample lines is 47352788.

The test results are as follows:

| BUF_SIZE | The elapsed time of Hadoop FS | The elapsed time of local|
|-----------|---------------------------------|--------------------------|
| 1M | 7531287724ns | 5878985850ns |
| 2M | 7434749749ns | 6601452869ns |
| 4M | 7262694142ns | 6601452869ns |

Conclusion:
We can use Hdfsfilereader to write local files instead of Localfilereader

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the LocalFileReader and HDFSFileReader implementations and review HadoopAndLocalReadTest, especially testHadoopReadLocal and testReadLocal. Compare their local-file behavior and benchmark assumptions, then define completion as a single reader approach that preserves the required reading behavior and passes the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.