gagolews / gagolews/stringi

Encoding sleuthing function

Open
#189 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
318
Forks
48
PR merge metrics
No merged PRs in 30d

Description

Sometimes you know what a character should be, and you know how it's stored in a file, and you want to figure out what encoding it might have been (e.g. https://github.com/hadley/readr/issues/252). It would be helpful to have something to help automate the search, e.g.

``` R
x <- "Ü"
encs <- names(stri_enc_list())
conv <- lapply(encs, function(to) stri_conv(x, "UTF-8", to, to_raw = TRUE)[[1]])
names(conv) <- encs
Filter(function(x) identical(x[[1]], charToRaw("\x86")), conv)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the R example with stri_enc_list() and stri_conv(), including the target raw byte \x86. Define an interface that automates this encoding search and verify that it identifies matching encodings without requiring the caller to write the lapply and Filter steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
internationalization
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.