WheatonCS / WheatonCS/Lexos

Provide access to tokenisation options in visualisation tools

Open
#533 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement Tokenization Vizualization
Dominant language
Python
Stars
123
Forks
20
PR merge metrics
No merged PRs in 30d

Description

Right now, Word Cloud, Multicloud, and BubbleViz use the default word tokenisation settings. If you want to show clouds of bigrams, for instance, you have to go to Tokenize/Count regenerate the table, and then return to the visualisation tool.

Tools like Clustering allow the user to set tokenisation options on the same page, so we should probably add the same "Advanced Options" settings to the visualisation tool templates. I see the main problem with this to be one of screen real estate.

A slightly related issue is how the tokenisation settings are changed. At the moment, you can go to Clustering and set Lexos to tokenise using bi-grams. This is nearly instantaneous for reasonable-sized corpora, even with the clustering. You can then go to BubbleViz and view the bi-grams. But, if you do the same thing in Tokenizer, it takes a long time for the table to be generated. This may be an argument for adding an option to re-tokenise without reloading the table to the screen.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the Word Cloud, Multicloud, and BubbleViz visualisation tool templates alongside Clustering's existing Advanced Options settings. Determine how tokenisation options can fit into those pages and whether re-tokenisation should avoid reloading the table; the work is done when the visualisation tools expose the intended options and support the expected tokenisation behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.