crytic / crytic/crytic-compile

Add support for solcjs (npm-installed Solidity compiler) for cross-architecture compatibility

Open
#611 0 comments 0 reactions 0 assignees View on GitHub
cross-platform enhancement
Dominant language
Python
Stars
197
Forks
97
PR merge metrics
No merged PRs in 30d

Description

# Support for solcjs (npm-installed Solidity compiler) in crytic-compile

## Background

The Solidity compiler ecosystem has two main distribution methods:
1. **Native binaries** (`solc`) - Platform-specific executables distributed by the Ethereum Foundation
2. **JavaScript/WASM version** (`solcjs`) - Cross-platform compiler distributed via npm

Currently, crytic-compile only supports native `solc` binaries and fails when encountering `solcjs`, limiting its use in cross-architecture environments where native binaries are unavailable.

## The Problem

### Architecture Compatibility Challenge

Native Solidity binaries have limited architecture support:
- **x86_64**: Full support for all versions
- **ARM64 (Apple Silicon, AWS Graviton, etc.)**: Limited or no support for older versions
- **Other architectures**: Generally unsupported

This creates problems for:
- Developers on Apple Silicon Macs
- CI/CD pipelines running on ARM64 instances
- Docker containers targeting multiple architectures
- Projects requiring older Solidity versions (e.g., 0.4.x) on non-x86_64 platforms

### Current Failure Mode

When crytic-compile encounters `solcjs` instead of native `solc`, it fails with:
```
crytic_compile.platform.exceptions.InvalidCompilation: Invalid solc compilation /usr/lib/node_modules/solc/soljson.js:1
var Module;if(!Module)Module=...
```

This happens because:
1. `solcjs` is actually a Node.js script, not a native binary
2. crytic-compile uses `subprocess.Popen` with `executable=shutil.which(cmd[0])` to invoke `solc`
3. Without a shell, the Node.js shebang (`#!/usr/bin/env node`) isn't processed
4. The JavaScript content is returned as stdout instead of being executed

### Affected Code Locations

The issue manifests in multiple places:
- `crytic_compile/platform/solc.py:385`: `subprocess.Popen(..., executable=shutil.which(cmd[0]))`
- `crytic_compile/platform/solc.py:556-565`: Similar subprocess invocations
- `crytic_compile/utils/subprocess.py:35-50`: Common subprocess utility

## Proposed Solution

### Option 1: Add Native solcjs Support (Recommended)

Create a new platform type `SolcJS` that:
1. Detects when `solcjs` is available instead of `solc`
2. Handles the different CLI interface (solcjs uses different flags than solc)
3. Translates between command-line formats

**Implementation approach:**
```python
class SolcJS(AbstractPlatform):
NAME = "solcjs"
TYPE = Type.SOLCJS

@staticmethod
def is_supported(target: str, **kwargs: str) -> bool:
# Check if solcjs is available and target is .sol file
solcjs_path = shutil.which("solcjs")
if solcjs_path and target.endswith(".sol"):
# Verify it's actually solcjs not native solc
return _is_nodejs_script(solcjs_path)
return False

def compile(self, crytic_compile: "CryticCompile", **kwargs: str) -> None:
# Translate solc arguments to solcjs format
# e.g., --combined-json abi,bin → --abi --bin + manual JSON assembly
# Invoke with proper Node.js execution
```

**Benefits:**
- Clean abstraction aligned with crytic-compile's architecture
- Automatic fallback when native solc unavailable
- Transparent to downstream tools (Slither, Manticore, etc.)

### Option 2: Modify Existing Solc Platform

Enhance the existing `Solc` platform to detect and handle both native and JS versions:

```python
def _run_solc(...):
# Check if solc is actually solcjs
if _is_solcjs(solc_path):
return _run_solcjs(...)
else:
# Existing native solc logic
```

**Benefits:**
- Smaller change footprint
- Single platform for all direct Solidity compilation

**Drawbacks:**
- Mixes two different compiler interfaces in one class
- More complex error handling

### Option 3: Fix subprocess invocation

Modify subprocess calls to use shell=True or explicitly invoke through Node.js:

```python
# Instead of:
subprocess.Popen(cmd, executable=shutil.which(cmd[0]))

# Use:
subprocess.Popen(cmd, shell=True) # Let shell handle shebang
# OR
subprocess.Popen(["node", solc_path] + cmd[1:]) # Explicit Node.js
```

**Drawbacks:**
- Security implications of shell=True
- Doesn't address CLI incompatibilities between solc and solcjs

## CLI Differences to Handle

The main differences between `solc` and `solcjs`:

| Feature | solc | solcjs |
|---------|------|--------|
| Combined JSON | `--combined-json abi,bin` | Not supported (must use `--abi --bin` separately) |
| Output format | Direct JSON output | Separate files or stdout |
| Remappings | `--remap` | Different format |
| Import paths | `--allow-paths` | `--base-path` and `--include-path` |

## Testing Requirements

1. **Unit tests**: Mock solcjs responses for different Solidity versions
2. **Integration tests**: Test with actual npm-installed solcjs
3. **Cross-platform CI**: Test on x86_64 and ARM64 architectures
4. **Version compatibility**: Test with solcjs versions matching solc 0.4.x through 0.8.x

## Migration Path

1. **Phase 1**: Implement solcjs support behind a feature flag
2. **Phase 2**: Auto-detect and use solcjs when native solc unavailable
3. **Phase 3**: Document usage and limitations
4. **Phase 4**: Consider deprecating workarounds in downstream projects

## Alternative Workarounds (Current State)

Projects currently work around this limitation by:
1. Creating wrapper scripts that translate between solc and solcjs
2. Using Docker images with pre-installed native binaries (x86_64 only)
3. Skipping tests on unsupported architectures
4. Using remote compilation services

None of these are ideal solutions.

## Related Issues

- solc-select doesn't support ARM64 Linux binaries
- Foundry increasingly used in projects, which has better cross-platform support
- Growing adoption of ARM64 in cloud (AWS Graviton) and development (Apple Silicon)

## Impact

Adding solcjs support would:
- Enable crytic-compile on all architectures
- Remove a significant barrier for Apple Silicon developers
- Reduce Docker image complexity
- Enable older Solidity version support on modern hardware
- Improve CI/CD flexibility with ARM64 instances

## Recommended Approach

We recommend **Option 1** (Native solcjs Support) because:
1. It aligns with crytic-compile's existing architecture
2. Provides clean separation of concerns
3. Allows for proper testing and maintenance
4. Enables gradual rollout and testing

## Next Steps

1. Validate the proposed approach with maintainers
2. Implement prototype of SolcJS platform
3. Test with real-world projects using various Solidity versions
4. Document limitations and usage
5. Coordinate with downstream projects for testing

## Code References

Key files that need modification:
- `/crytic_compile/platform/solcjs.py` (new file)
- `/crytic_compile/platform/all_platforms.py:19` (add import)
- `/crytic_compile/platform/types.py` (add SOLCJS type)
- `/crytic_compile/crytic_compile.py:600-618` (platform selection logic)

## Use Case Example

This issue was discovered while trying to run Manticore tests in Docker on ARM64. The Dockerfile installs solcjs via npm since native solc 0.4.25 isn't available for ARM64, but crytic-compile fails to recognize and properly execute solcjs, making cross-architecture testing impossible.

## Conclusion

Supporting solcjs would significantly improve crytic-compile's usability across different architectures and environments. While there are multiple approaches, adding proper solcjs support as a new platform provides the cleanest and most maintainable solution.

Contributor guide

Open the contributing guide

Research direction

Start by reading crytic_compile/platform/solc.py, crytic_compile/utils/subprocess.py, and the platform registration and selection paths in all_platforms.py, types.py, and crytic_compile.py. Run the existing compiler tests before comparing native solc and npm-installed solcjs behavior. Done means solcjs is detected and invoked with compatible arguments, with unit, integration, version, and architecture coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, python, solidity
Domain
build-system, compilers, devtools
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.