apache / apache/hugegraph-toolchain
Merge LocalFileReader and HDFSFileReader
- Dominant language
- Java
- Stars
- 121
- Forks
- 125
- Avg merge
- 10d 10h
- Merged PRs (30d)
- 3
Description
The code is as follows:
```
public class HadoopAndLocalReadTest {
private static final long BUF_SIZE = 4 * 1024 *1024;
@Test
public void testHadoopReadLocal() throws IOException {
long startTime=System.nanoTime();
Path file = new Path("file:///Downloads/index-00001");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
FSDataInputStream getIt = fs.open(file);
BufferedReader d = new BufferedReader(new InputStreamReader(getIt),
(int) BUF_SIZE);
String lineTxt = null;
while ((lineTxt = d.readLine()) != null) {
}
long endTime=System.nanoTime();
System.out.println("The elapsed time of Hadoop: " + (endTime - startTime) + "ns");
d.close();
fs.close();
}
@Test
public void testReadLocal() throws IOException {
long startTime=System.nanoTime();
String filePath = "/Downloads/index-00001";
File file = new File(filePath);
InputStreamReader read = new InputStreamReader(
new FileInputStream(file));
BufferedReader bufferedReader = new BufferedReader(read, (int) BUF_SIZE);
String lineTxt = null;
while ((lineTxt = bufferedReader.readLine()) != null) {
}
long endTime=System.nanoTime();
System.out.println("The elapsed time of local : " + (endTime - startTime) + "ns");
bufferedReader.close();
read.close();
}
}
```
The number of test sample lines is 47352788.
The test results are as follows:
| BUF_SIZE | The elapsed time of Hadoop FS | The elapsed time of local|
|-----------|---------------------------------|--------------------------|
| 1M | 7531287724ns | 5878985850ns |
| 2M | 7434749749ns | 6601452869ns |
| 4M | 7262694142ns | 6601452869ns |
Conclusion:
We can use Hdfsfilereader to write local files instead of Localfilereader
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the LocalFileReader and HDFSFileReader implementations and review HadoopAndLocalReadTest, especially testHadoopReadLocal and testReadLocal. Compare their local-file behavior and benchmark assumptions, then define completion as a single reader approach that preserves the required reading behavior and passes the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100