wish: WARCHdfsBolt with CDX index
- Dominant language
- Java
- Stars
- 995
- Forks
- 292
- Avg merge
- 2d 49m
- Merged PRs (30d)
- 62
Description
StormCrawler allows to filter web pages and archive them into WARC archives, as follows:
```
WARCHdfsBolt warcbolt = (WARCHdfsBolt) new WARCHdfsBolt().withFileNameFormat(fileNameFormat);
TopologyBuilder builder = new TopologyBuilder();
builder.setBolt("warc", warcbolt, numWorkers)
.localOrShuffleGrouping("parse", WarcStreamName)
.localOrShuffleGrouping("tika", WarcStreamName);
```
Would it be possible to create a [CDX index](https://iipc.github.io/warc-specifications/specifications/cdx-format/cdx-2015/) (or JCDX index) for the WARC archives at the same time?
Contributor guide
Research direction
Start with WARCHdfsBolt and the linked CDX format specification, then compare the existing WARC-writing flow with the metadata required for a CDX or JCDX index. Define how the index should be produced alongside the WARC archives and what output validation would demonstrate completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100