typesense / typesense/typesense

Inaccurate example in stemming docs – describes lemmatization instead

Open
#2,449 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
26.6k
Forks
973
Avg merge
18h 45m
Merged PRs (30d)
4

Description

I noticed that the current documentation for stemming seems to conflate stemming with lemmatization.

Specifically, the following example is given:

“running”, “runner”, “ran”, “runs” are all reduced to the root word “run”.

However, this behavior is characteristic of lemmatization, which uses contextual and morphological analysis to reduce words to their dictionary base form.

In contrast, stemming is a rule-based process that typically does not reduce irregular forms like "ran" to "run".

Hence, the actual current behavior likely is closer to:

  • "running" → "run"
  • "runs" → "run"
  • "ran" → "ran" (unchanged)
  • "runner" → "runner" or possibly "run" depending on stemmer, but not reliably

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked stemming documentation at typesense.org/docs/29.0/api/stemming.html and verify the stated examples against the documented or implemented stemming behavior. Update the example so it accurately distinguishes stemming from lemmatization and reflects the actual treatment of “running,” “runs,” “ran,” and “runner.”

Written by the indexing model from the issue text.

Assessment

Domain
documentation, search
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.