apache / apache/lucene

Fix directory deletion in ExtractReuters, recover from errors [LUCENE-6073]

Open
#7,135 4 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:benchmark type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

ExtractReuters in the benchmark module currently fails because it currently creates the output directory, and then calls `IOUtils.rm` on it (which will remove all files in it as well as removes the output directory itself). This is to fix this behaviour.

While I was at it, I also added a bit more logging in case of file errors (the download I had some bad data) and made the task recover in case of issues with one file.

---
Migrated from [LUCENE-6073](https://issues.apache.org/jira/browse/LUCENE-6073) by Ramkumar Aiyengar (@andyetitmoves), updated Nov 30 2014

Contributor guide

Open the contributing guide

Research direction

Start at the ExtractReuters benchmark entry point and inspect how it prepares the output directory before calling IOUtils.rm. Review the file-error handling and logging paths described in the issue, then run the relevant benchmark tests or task checks; done means directory extraction no longer removes its output directory and one bad file does not stop the task.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.