Add Chinese Word Segmentation Analyzer with Ansj implementation [LUCENE-6111]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
When I use mahout-0.9 depending on lucene-4.6 to run Kmeans clustering algorithm, I find that the default word segmentation analyzer class named 'org.apache.lucene.analysis.standard.StandardAnalyzer' is very ugly, only single word could be splitted.However, ansj Chinese word segmentation tool is widely used in Chinese document-tokenizer, and I am willing to add it to support lucene.
---
Migrated from [LUCENE-6111](https://issues.apache.org/jira/browse/LUCENE-6111) by deyinchen
Contributor guide
Research direction
Start by reviewing org.apache.lucene.analysis.standard.StandardAnalyzer and the migrated LUCENE-6111 request to understand the existing analysis entry point. Confirm how Ansj Chinese word segmentation should be integrated and what analyzer behavior and tests would demonstrate support.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100