zopencommunity / zopencommunity/gitport

Using .gitattributes to encode file as IBM-1047 results in some characters being converted incorrectly

Open
#63 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
10
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Description

The way I have it set up is that the file is stored as UTF-8 in git, and I'm using .gitattributes to set the working tree encoding to ibm-1047. When cloning the repo, git converts and tags the file as IBM-1047. The issue is that for some characters such as ® (the registered symbol), it appears as ▒ after the conversion.

Reproduce

  1. Create a new repository.
  2. Create file.txt containing the ® character.
  3. Create a .gitattributes file containing either file.txt zos-working-tree-encoding=ibm-1047 git-encoding=iso8859-1 or file.txt working-tree-encoding=ibm-1047
  4. Clone the repository.
  5. Read file.txt using vim or cat. It will display ▒ instead of ®

Additional info

Upon examination of the file with a hex editor, it appears that it's converting the ® character from C2 AE to AF which results in it being unreadable. Whereas for it to be readable, it would have to be 62 AF.

This is consistent with the behavior when using iconv to convert from UTF-8 to IBM-1047.

Whereas converting from ISO8859-1 to IBM-1047 seems to result in the correct conversion.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproduced .gitattributes cases for file.txt and compare the UTF-8-to-IBM-1047 results with the ISO-8859-1-to-IBM-1047 results described in the issue. Check the relevant git encoding behavior and use the iconv comparison as a reference; done means cloning and reading file.txt preserves ® rather than displaying ▒.

Written by the indexing model from the issue text.

Assessment

Tech stack
git
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.