(v0.8.26) Contract code changes with additional files in compilation
- Dominant language
- C++
- Stars
- 25.7k
- Forks
- 6.2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 29
Description
Here I am with another instance of #14494 and #14250.
Submitting this issue because these are supposed to be fixed in the latest versions but I encountered this originally in v0.8.26, and was able to also reproduce in v0.8.28
## JSON Inputs
There are two instances (ie. std JSONs) of the same contract. One with extra files and one with only the files used by the compilation.
Below are the outputs with 0.8.26+commit.8a97fa7a. I haven't noticed any differences across binaries solc-js vs Darwin
- `etherscan-response-input.json` (Larger JSON): This is the full JSON with extra files, called "etherscan-response" because this is from Etherscan and its the content of the response
- [11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input.json](https://github.com/user-attachments/files/18282319/11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input.json)
- `sourcify-etherscan-import-input.json` (Minimal JSON): This is the minimal JSON because we first import the file from Etherscan, generate the metadata by compiling the Larger JSON, and compile with the JSON based on this metadata. Since metadata.json generated strips away the source files not used by this contract, this creats a minimal JSON (expectedly).
- [11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-input.json](https://github.com/user-attachments/files/18282329/11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-input.json)
## Input differences
Here's the output of the script that checks for source file differences between the two JSON inputs:
Script Output
```bash
$ node compare-sources.js 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input.json 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-input.json
Keys in 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input.json but not in 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-input.json: [
'@openzeppelin/contracts/access/AccessControl.sol',
'@openzeppelin/contracts/access/Ownable.sol',
'@openzeppelin/contracts/interfaces/IERC5267.sol',
'@openzeppelin/contracts/proxy/beacon/BeaconProxy.sol',
'@openzeppelin/contracts/proxy/beacon/IBeacon.sol',
'@openzeppelin/contracts/proxy/beacon/UpgradeableBeacon.sol',
'@openzeppelin/contracts/proxy/Clones.sol',
'@openzeppelin/contracts/proxy/ERC1967/ERC1967Utils.sol',
'@openzeppelin/contracts/proxy/Proxy.sol',
'@openzeppelin/contracts/token/ERC20/ERC20.sol',
'@openzeppelin/contracts/token/ERC20/extensions/IERC20Permit.sol',
'@openzeppelin/contracts/token/ERC20/utils/SafeERC20.sol',
'@openzeppelin/contracts/token/ERC721/ERC721.sol',
'@openzeppelin/contracts/token/ERC721/extensions/ERC721Burnable.sol',
'@openzeppelin/contracts/token/ERC721/extensions/IERC721Metadata.sol',
'@openzeppelin/contracts/token/ERC721/IERC721.sol',
'@openzeppelin/contracts/token/ERC721/IERC721Receiver.sol',
'@openzeppelin/contracts/utils/Address.sol',
'@openzeppelin/contracts/utils/Context.sol',
'@openzeppelin/contracts/utils/cryptography/EIP712.sol',
'@openzeppelin/contracts/utils/cryptography/MessageHashUtils.sol',
'@openzeppelin/contracts/utils/introspection/ERC165.sol',
'@openzeppelin/contracts/utils/ShortStrings.sol',
'@openzeppelin/contracts/utils/StorageSlot.sol',
'contracts-exposed/LoanNFT/LoanXNFT.sol',
'contracts-exposed/LoanPaymentGateway/ImplLoanPaymentGateway.sol',
'contracts-exposed/LoanPaymentGateway/LoanPaymentGatewayFactory.sol',
'contracts-exposed/LoanX/LoanX.sol',
'contracts-exposed/mocks/MockedUSDC.sol',
'contracts-exposed/mocks/MockedUSDT.sol',
'contracts-exposed/SecurityToken/BeaconFactory/BeaconFactory.sol',
'contracts-exposed/SecurityToken/SecurityToken/modules/access/AccessControlModule.sol',
'contracts-exposed/SecurityToken/SecurityToken/modules/controllers/ERC20ControllerModule/ERC20ControllerModule.sol',
'contracts-exposed/SecurityToken/SecurityToken/modules/controllers/SignatureVerifier/SignatureVerifier.sol',
'contracts-exposed/SecurityToken/SecurityToken/modules/core/ERC20BaseModule/ERC20BaseModule.sol',
'contracts-exposed/SecurityToken/SecurityToken/modules/core/ERC20FrozenModule/ERC20FrozenModule.sol',
'contracts-exposed/SecurityToken/SecurityToken/modules/core/ERC20IssueModule/ERC20IssueModule.sol',
'contracts-exposed/SecurityToken/SecurityToken/modules/core/ERC20RedeemModule/ERC20RedeemModule.sol',
'contracts-exposed/SecurityToken/SecurityToken/modules/core/InfoModule/InfoModule.sol',
'contracts-exposed/SecurityToken/SecurityToken/modules/core/PauseModule/PauseModule.sol',
'contracts-exposed/SecurityToken/SecurityToken/modules/core/SnapshotModule/ERC20SnapshotModule.sol',
'contracts-exposed/SecurityToken/SecurityToken/modules/SecurityTokenBase.sol',
'contracts-exposed/SecurityToken/SecurityToken/SecurityTokenProxy.sol',
'contracts-exposed/SecurityTokenPaymentGateway/BaseSecurityTokenPaymentGatewayFactory.sol',
'contracts-exposed/SecurityTokenPaymentGateway/ImplementationEU.sol',
'contracts-exposed/SecurityTokenPaymentGateway/ImplementationUS.sol',
'contracts-exposed/SecurityTokenPaymentGateway/library/CloneWithStorage.sol',
'contracts-exposed/SecurityTokenPaymentGateway/SecurityTokenPaymentGatewayFactory.sol',
'contracts/LoanNFT/interfaces/ILoanXNFT.sol',
'contracts/LoanNFT/LoanXNFT.sol',
'contracts/LoanPaymentGateway/ImplLoanPaymentGateway.sol',
'contracts/LoanPaymentGateway/interfaces/IImplLoanPaymentGateway.sol',
'contracts/LoanPaymentGateway/interfaces/ILoanPaymentGatewayFactory.sol',
'contracts/LoanPaymentGateway/interfaces/ILoanPaymentGatewayFactoryErrors.sol',
'contracts/LoanPaymentGateway/LoanPaymentGatewayFactory.sol',
'contracts/LoanX/interfaces/ILoanX.sol',
'contracts/LoanX/interfaces/ILoanXErrors.sol',
'contracts/LoanX/interfaces/ILoanXStructs.sol',
'contracts/LoanX/LoanX.sol',
'contracts/mocks/MockedUSDC.sol',
'contracts/mocks/MockedUSDT.sol',
'contracts/SecurityToken/BeaconFactory/BeaconFactory.sol',
'contracts/SecurityToken/BeaconFactory/interfaces/IBeaconFactory.sol',
'contracts/SecurityToken/BeaconFactory/interfaces/IBeaconFactoryErrors.sol',
'contracts/SecurityTokenPaymentGateway/BaseSecurityTokenPaymentGatewayFactory.sol',
'contracts/SecurityTokenPaymentGateway/ImplementationEU.sol',
'contracts/SecurityTokenPaymentGateway/ImplementationUS.sol',
'contracts/SecurityTokenPaymentGateway/interfaces/IBaseSecurityTokenPaymentGatewayFactory.sol',
'contracts/SecurityTokenPaymentGateway/interfaces/IBaseSecurityTokenPaymentGatewayFactoryErrors.sol',
'contracts/SecurityTokenPaymentGateway/interfaces/IImplementationEU.sol',
'contracts/SecurityTokenPaymentGateway/interfaces/IImplementationUS.sol',
'contracts/SecurityTokenPaymentGateway/interfaces/ISecurityTokenPaymentGatewayFactory.sol',
'contracts/SecurityTokenPaymentGateway/library/CloneWithStorage.sol',
'contracts/SecurityTokenPaymentGateway/SecurityTokenPaymentGatewayFactory.sol'
]
Keys in 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-input.json but not in 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input.json: []
```
The JS script
```js
const fs = require("fs");
// Function to read and parse JSON files
function readJSONFile(filePath) {
try {
const data = fs.readFileSync(filePath, "utf8");
return JSON.parse(data);
} catch (error) {
console.error(`Error reading or parsing file ${filePath}:`, error);
process.exit(1);
}
}
// Get file paths from command-line arguments
const [file1Path, file2Path] = process.argv.slice(2);
if (!file1Path || !file2Path) {
console.error("Please provide paths for both files as arguments.");
process.exit(1);
}
// Read and parse the JSON files
const file1 = readJSONFile(file1Path);
const file2 = readJSONFile(file2Path);
// Extract the keys of the sources objects
const file1Keys = Object.keys(file1.sources);
const file2Keys = Object.keys(file2.sources);
// Find keys that are in file1 but not in file2
const inFile1NotInFile2 = file1Keys.filter((key) => !file2Keys.includes(key));
// Find keys that are in file2 but not in file1
const inFile2NotInFile1 = file2Keys.filter((key) => !file1Keys.includes(key));
// Output the results
console.log(`Keys in ${file1Path} but not in ${file2Path}:`, inFile1NotInFile2);
console.log(`Keys in ${file2Path} but not in ${file1Path}:`, inFile2NotInFile1);
```
## Bytecode differences
Compiled runtime bytecode of the Larger JSON:
[11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-output-runtime-bytecode.txt](https://github.com/user-attachments/files/18282280/11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-output-runtime-bytecode.txt)
Compiled runtime bytecode of the Minimal JSON:
[11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-runtime-bytecode.txt](https://github.com/user-attachments/files/18282369/11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-runtime-bytecode.txt)
Even though they are mostly identical, the bytecodes are also different in lengths
- Larger JSON bytecode: 30000 characters
- Minimal JSON bytecode: 30016 characters
Taking the bytecode diff:
```
git diff --word-diff --word-diff-regex=. 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-output-runtime-bytecode.txt 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-import-from-etherscan-output-runtime-bytecode.txt
```
Some screenshots from the diff.
Middle:
Only the last part:
In the Middle example, you can see the pattern `3` being replaced by `b`
In the Last Part, there are longer additions in green (hence the difference in lengths). You can see the metadata hashes are identical.
## v0.8.28
I also reproduced this with 0.8.28. The Minimal JSON compiles directly with 0.8.28, although the Larger JSON has strict versions `pragma solidity 0.8.26;` so I need to manually set them to `pragma solidity 0.8.28;`
Larger JSON with replaced 0.8.28:
[11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input-0.8.28.json](https://github.com/user-attachments/files/18282360/11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input-0.8.28.json)
Minimal JSON is identical
Larger JSON 0.8.28 runtime bytecode:
[11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-output-0.8.28-runtime-bytecode.txt](https://github.com/user-attachments/files/18282378/11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-output-0.8.28-runtime-bytecode.txt)
Minimal JSON 0.8.28 runtime bytecode:
[11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-0.8.28-runtime-bytecode.txt](https://github.com/user-attachments/files/18282380/11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-0.8.28-runtime-bytecode.txt)
Diff command
```
git diff --word-diff --word-diff-regex=. 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-output-0.8.28-runtime-bytecode.txt 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-0.8.28-runtime-bytecode.txt
```
Similar noticable patterns:
Middle:
Ending:
The bytecode lengths are same as before (30000 vs 30016 characters). The bytecode between same JSONs but different versions ("Larger JSON 0.8.26" vs. "Larger JSON 0.8.28") are different as expected. But from what I can see the shifts are the same, ie. the diff screenshots above and here are almost identical.
Notice the metadata hashes are again identical. One might have expected the metadata hash to change since we touched `pragma` statements, however we only touched the files that are not supposed to be used in the compilation. Therefore the metadata hashes remain the same expectedly.
For convenience I'm also adding the command I used to extract the runtime bytecode from the JSON outputs:
```bash
cat 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-0.8.28.json | jq '.contracts.["contracts/SecurityToken/SecurityToken/SecurityTokenProxy.sol"].SecurityTokenProxy.evm.deployedBytecode.object' > 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-0.8.28-runtime-bytecode.txt
```
Contributor guide
Research direction
Start by compiling the attached etherscan-response and sourcify-etherscan-import JSON inputs with solc 0.8.26 and 0.8.28, then use compare-sources.js and the provided git diff commands to compare sources and runtime bytecode. Done means the differing behavior is isolated and the compiler no longer produces unexpected bytecode differences when extra source files are present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, solidity
- Domain
- blockchain, compilers
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100