argotorg / argotorg/solidity

Restrict source files to ASCII, only allowing unicode via pragma.

Open
#10,607 3 comments 0 reactions 0 assignees View on GitHub
breaking change :warning: language design :rage4: low effort medium impact
Dominant language
C++
Stars
25.7k
Forks
6.2k
Avg merge
2d 19h
Merged PRs (30d)
29

Description

I just want to bring this up for debate (for a future breaking release).

I'm not entirely convinced and sure that it's feasible to actually safely implement all unicode quirks with mechanisms like https://github.com/ethereum/solidity/pull/10326 (@cameel seems similarly sceptical, if I understood the comments correctly - please correct me if I'm wrong).

Based on that, I'd propose the following:
Any non-ASCII character in any source file is an error by default.
However, it's possible to add ``pragma source-encoding utf-8;`` (or something similar - only supported values would be ``ascii`` and ``utf-8``) to allow unicode characters (*after* that pragma).

The advantage of this is that this pragma would be a very clear hint to any auditor that they need to look out for unicode attacks.

This wouldn't mean that we should not still go for trying to extend unicode support, e.g. like in https://github.com/ethereum/solidity/pull/10326, which we definitely *should* do for inclusiveness reasons alone, but it would decrease the danger in all of this.

However, one can easily argue against this:
- The fact that *not* using the pragma and restricting to ASCII might be preferred by auditors, would work *against* inclusiveness.
- Still supporting utf-8 sources with the pragma does not diminish our need of implementing proper support.

Still, I don't think it wise to try implementing "proper" unicode support ourselves. If we really want it, I think we should fall back on an external implementation like libicu (even though this is one of the largest most annoying dependencies I have ever seen projects depending on - but it's not without reason that it is - proper unicode support is insanely complex).

EDIT: also, even though not relevant to this issue, note that if we used a complete external unicode implementation, we might also be able to safely allow unicode identifiers again, following e.g. Unicode Standard Annex 31 like C++ in http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1949r5.html

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named. Start by reviewing the Unicode-handling concerns and PR 10326, then examine whether libicu or a source-encoding pragma can define safe, testable behavior; done requires an agreed design rather than the current debate.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, solidity
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.