google / google/go-jsonnet

std.base64 fails on unicode strings with codepoints > 255 (emoji, CJK, etc.)

Open
#887 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.8k
Forks
263
PR merge metrics
No merged PRs in 30d

Description

## Bug Description

`std.base64()` fails when the input string contains characters with codepoints greater than 255, such as emoji, CJK characters, and other non-Latin scripts.

## Reproduction

```jsonnet
std.base64("🎉")
```

## Expected Behavior

```json
"8J+OiQ=="
```

(This is the correct base64 encoding of the UTF-8 bytes `f0 9f 8e 89` for 🎉)

## Actual Behavior

```
RUNTIME ERROR: base64 encountered invalid codepoint value in the array (must be 0 <= X <= 255), got 127881
```

## More Examples

```jsonnet
std.base64("你好") // fails - Chinese characters
std.base64("مرحبا") // fails - Arabic characters
std.base64("héllo") // works - all codepoints <= 255
std.base64("hello") // works - ASCII
```

## Version

```
$ jsonnet --version
Jsonnet commandline interpreter (Go implementation) v0.22.0
```

## Analysis

The implementation appears to convert the string to an array of Unicode codepoints and then try to treat each codepoint as a byte. This fails for any character with a codepoint > 255.

The correct behavior should be:
1. Encode the string as UTF-8 bytes
2. Base64-encode the resulting byte array

sjsonnet handles this correctly:
```jsonnet
std.base64("🎉") // returns "8J+OiQ==" (correct)
std.base64("你好") // returns "5L2g5aW9" (correct)
```

Verification that the expected output is correct:
```bash
echo -n "🎉" | base64
# Output: 8J+OiQ==
```

Contributor guide

Open the contributing guide

Research direction

Start at the std.base64 entry point described in the issue and reproduce the 🎉, CJK, and Arabic examples with the Go implementation. Trace how the string reaches base64 encoding and compare it with the stated UTF-8 byte expectations. Done means non-ASCII strings return the expected base64 values without the invalid-codepoint error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
compilers
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.