gchq / gchq/CyberChef

Bug report: Output shows raw format while data is UTF-8 thus showing incorrect data in UI

Open
#1,729 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
35.8k
Forks
4.1k
Avg merge
2d 26m
Merged PRs (30d)
33

Description

**Describe the bug**
Output shows raw format while data is UTF-8 thus showing incorrect data in UI. Need to manually change Output Format Encoding to UTF-8 to see correct data. This happens with or without a recipe to process the data

Even trying to force we encode and decode text doesn't change this behavior.

**To Reproduce**
Steps to reproduce the behavior or a link to the recipe / input used to cause the bug:

Scenario 1:
1. Leave everything on default (input and output on raw, no recipe)
2. In Input open a UTF-8 file with some UTF-8 specific content (in my test case a Russian name)
3. In Output look at content and content is shown incorrectly
4. Change manually to UTF-8. Now shows correctly

Scenario 2:
1. Leave everything on default (input and output on raw, no recipe)
2. In Input open a UTF-8 file with some UTF-8 specific content (in my test case a Russian name)
3. Process recipe that includes decode and encode text step to force UTF-8
4. In Output look at content and content is shown incorrectly
6. Change manually to UTF-8. Now shows correctly

Scenario 3:
1. Leave everything on default (input and output on raw, no recipe)
2. In Input open a UTF-8 file with some UTF-8 specific content (in my test case a Russian name) that was encoded (in this case base64)
3. Process recipe that includes decodes the base64
4. In Output look at content and content is shown incorrectly (even if we try to add a encode step_
5. Change manually to UTF-8. Now shows correctly

**Expected behavior**
When file is processed using a recipe and the format is a specific encoding automatically adjust output character encoding to match correct format. If this is not possible have new special "flow" or "util" step to force specific encoding in UI.
Tried to force it by using decode text and encode text but that didn't work.
I also saved the resulting output and then opened that without any recipe and I also get inconsistent behavior. You can see the result in the below screenshots. The input show the data in UTF-8 although input format encoding shows Raw Bytes and the output shows the data as raw bytes and the output format encoding shows Raw Bytes

**Screenshots**
![Show opened file after save of output](https://github.com/gchq/CyberChef/assets/103523896/150ca992-cec6-4ca6-9f48-3f1635ced143)

![Show opened file after save of output - Update to utf8](https://github.com/gchq/CyberChef/assets/103523896/0d592895-542a-4d08-b0c8-6fd3fa915bb1)

**Desktop (if relevant, please complete the following information):**
- OS: Windows
- Browser: Edge version 121.0.2277.83 (also tested with Chrome)
- CyberChef version: 10.8.0

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the three scenarios in the CyberChef UI using a UTF-8 file, including the base64 recipe, and compare the input and output format encoding controls. Trace the handling of raw bytes, UTF-8 display, and recipe output. Done means processed UTF-8 data is displayed correctly without manually changing the output encoding, or the limitation is clearly surfaced to users.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
frontend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.