biolab / biolab/orange3-text

Automatic language detection in Corpus with language listed in output

Open
#983 3 comments 0 reactions 0 assignees View on GitHub
feast help wanted
Dominant language
Python
Stars
134
Forks
86
Avg merge
6d 5h
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe.**
I have a data file with abstracts from PhD theses in a column "ABSTRACT". Most abstracts are in English, but some are in Italian, and I want to be able to filter by language (e.g. using Select Rows). I get the impression that the latest versions of the Text add-on (>= 1.13) support automatic language detection, and in #916 it is suggested that this has now been incorporated in Corpus. However, from what I see, the language still has to be selected by the user, and there is no column in the output that lists the language.

**Describe the solution you'd like**
In the Language drop-down menu, add the option "guess" to the list of languages, and add a column "Language" to the output that lists the language of each row.

**Describe alternatives you've considered**
As I've learned that appearance of the word "il" (="the") distinguishes Italian text from English text, I inserted a Feature Constructor with the following assignment `Language := 'IT' if 'il' in ABSTRACT.lower().split() or ABSTRACT.lower().startswith('il') else 'EN'
` but it works for this specific case only and it's rather awkward.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the Corpus implementation and the Language drop-down handling in the Text add-on, then review issue #916 for the existing automatic detection work. Confirm how language values are represented before defining the output column and filtering behavior; done means a guess option detects each row's language and exposes it in Corpus output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.