[PROPOSAL] Markov sample separator
- Dominant language
- Go
- Stars
- 43
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
Markov Mask can be used on different samples:
- lists of words
- paragraphs
For list of words, we would want to read the file line by line (exemples: nameFR, pokemons, etc..)
For entire paragraphs, or text that can be spread over multiple lines.
## Proposal
In addition of the `separator` parameter that determine the way we split the text (word by word, character by character, etc..), we would want a parameter that helps the mask to understand the structure of the text:
- is it a list?
- is it paragraphs?
- is it something else?
Anyway, markov mask should have a default configuration in order not to make it unusable.
_Originally posted by @Baguettte in https://github.com/CGI-FR/PIMO/issues/81#issuecomment-1055246481_
Contributor guide
Research direction
Start by locating the Markov Mask implementation and the existing separator parameter handling. Clarify how list, paragraph, and other input structures should differ, choose the default behavior, and identify the relevant tests or add coverage for each supported structure before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100