std.base64 fails on unicode strings with codepoints > 255 (emoji, CJK, etc.)
- Dominant language
- Go
- Stars
- 1.8k
- Forks
- 263
- PR merge metrics
- No merged PRs in 30d
Description
## Bug Description
`std.base64()` fails when the input string contains characters with codepoints greater than 255, such as emoji, CJK characters, and other non-Latin scripts.
## Reproduction
```jsonnet
std.base64("🎉")
```
## Expected Behavior
```json
"8J+OiQ=="
```
(This is the correct base64 encoding of the UTF-8 bytes `f0 9f 8e 89` for 🎉)
## Actual Behavior
```
RUNTIME ERROR: base64 encountered invalid codepoint value in the array (must be 0 <= X <= 255), got 127881
```
## More Examples
```jsonnet
std.base64("你好") // fails - Chinese characters
std.base64("مرحبا") // fails - Arabic characters
std.base64("héllo") // works - all codepoints <= 255
std.base64("hello") // works - ASCII
```
## Version
```
$ jsonnet --version
Jsonnet commandline interpreter (Go implementation) v0.22.0
```
## Analysis
The implementation appears to convert the string to an array of Unicode codepoints and then try to treat each codepoint as a byte. This fails for any character with a codepoint > 255.
The correct behavior should be:
1. Encode the string as UTF-8 bytes
2. Base64-encode the resulting byte array
sjsonnet handles this correctly:
```jsonnet
std.base64("🎉") // returns "8J+OiQ==" (correct)
std.base64("你好") // returns "5L2g5aW9" (correct)
```
Verification that the expected output is correct:
```bash
echo -n "🎉" | base64
# Output: 8J+OiQ==
```
Contributor guide
Research direction
Start at the std.base64 entry point described in the issue and reproduce the 🎉, CJK, and Arabic examples with the Go implementation. Trace how the string reaches base64 encoding and compare it with the stated UTF-8 byte expectations. Done means non-ASCII strings return the expected base64 values without the invalid-codepoint error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100