Automatic language detection in Corpus with language listed in output
- Dominant language
- Python
- Stars
- 134
- Forks
- 86
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 1
Description
**Is your feature request related to a problem? Please describe.**
I have a data file with abstracts from PhD theses in a column "ABSTRACT". Most abstracts are in English, but some are in Italian, and I want to be able to filter by language (e.g. using Select Rows). I get the impression that the latest versions of the Text add-on (>= 1.13) support automatic language detection, and in #916 it is suggested that this has now been incorporated in Corpus. However, from what I see, the language still has to be selected by the user, and there is no column in the output that lists the language.
**Describe the solution you'd like**
In the Language drop-down menu, add the option "guess" to the list of languages, and add a column "Language" to the output that lists the language of each row.
**Describe alternatives you've considered**
As I've learned that appearance of the word "il" (="the") distinguishes Italian text from English text, I inserted a Feature Constructor with the following assignment `Language := 'IT' if 'il' in ABSTRACT.lower().split() or ABSTRACT.lower().startswith('il') else 'EN'
` but it works for this specific case only and it's rather awkward.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the Corpus implementation and the Language drop-down handling in the Text add-on, then review issue #916 for the existing automatic detection work. Confirm how language values are represented before defining the output column and filtering behavior; done means a guess option detects each row's language and exposes it in Corpus output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100