CGI-FR / CGI-FR/PIMO

[PROPOSAL] Markov sample separator

Open
#90 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
43
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Problem
Markov Mask can be used on different samples:
- lists of words
- paragraphs

For list of words, we would want to read the file line by line (exemples: nameFR, pokemons, etc..)
For entire paragraphs, or text that can be spread over multiple lines.

## Proposal

In addition of the `separator` parameter that determine the way we split the text (word by word, character by character, etc..), we would want a parameter that helps the mask to understand the structure of the text:
- is it a list?
- is it paragraphs?
- is it something else?

Anyway, markov mask should have a default configuration in order not to make it unusable.

_Originally posted by @Baguettte in https://github.com/CGI-FR/PIMO/issues/81#issuecomment-1055246481_

Contributor guide

Open the contributing guide

Research direction

Start by locating the Markov Mask implementation and the existing separator parameter handling. Clarify how list, paragraph, and other input structures should differ, choose the default behavior, and identify the relevant tests or add coverage for each supported structure before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.