biolab / biolab/orange3-text

Preprocess Text + Collocations: enable including n-grams

Open
#897 0 comments 0 reactions 0 assignees View on GitHub
enhancement meal
Dominant language
Python
Stars
134
Forks
86
Avg merge
6d 5h
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe.**
It is often desired to include only specific n-grams into the analysis. For example, n-grams based on PMI and some frequency metric. Currently, it is not possible to do this in Preprocess Text.

**Describe the solution you'd like**
Pass a list of selected words (n-grams) from Collocations to Preprocess Text. This could be of type Words. Use Words input for filtering.

**Describe alternatives you've considered**
Save the list with Save Data. However, we would still need to enable using n-grams in filtering (I don't think it currently works). And applying the filter only to n-grams, not 1-grams (or something).

**Additional context**
Words, such as "the European Union", "Jules Verne" or "cutting board" should be included as n-grams, rather than single tokens.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the Preprocess Text and Collocations widgets, including their Words inputs and filtering behavior. Trace how selected n-grams could pass from Collocations into Preprocess Text, then verify that multi-word terms are retained as n-grams while the requested filtering scope is respected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.