[Bug]Toolkit db mv may delete the source database after a copy failure, causing data loss

Start at the Toolkit db mv command and trace the Files.copy, source-deletion, symbolic-link, and status-reporting flow shown in the issue. Verify failure handling with a disposable database and isolated filesystem, then confirm that a copy failure preserves sources, cleans incomplete destination data, avoids move db done., and returns a non-zero status.

Bug Description

When Toolkit db mv copies database files, an IOException from an individual Files.copy call is printed but not propagated or recorded as an overall migration failure:

try {
  Files.copy(original, destination, StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
  spec.commandLine().getErr().println(e);
}

After copying, the command still attempts to delete the source directory. If deletion succeeds, it replaces the original path with a symbolic link to the destination:

if (FileUtils.deleteDir(p.original.toFile())) {
  Files.createSymbolicLink(p.original, p.destination);
}

If copying fails because the destination runs out of disk space or inodes, or because of another I/O error, the source database may be deleted and replaced with a symbolic link to an incomplete LevelDB/RocksDB database. The command may still print move db done. and exit with status 0.

The node may then fail to start and require recovery from a backup or a full resynchronization.

Environment

Network

N/A (offline maintenance tool)

Software Versions

OS: macOS 15 / Linux
JVM: Java 8 / Java 17
Git Commit: f8b05d40abc949fa588ab64d8fd8fd82845ebeed
Version: GreatVoyage-v4.8.2.1
Code: N/A

Expected Behavior

If any file copy fails, the command should:

• Preserve all source databases.
• Not create a symbolic link to an incomplete destination.
• Attempt to clean up incomplete destination data created by the migration.
• Not print move db done..
• Return a non-zero exit status.

Source directories should be deleted and replaced with symbolic links only after all databases have been copied successfully. A source deletion or symbolic-link creation failure should also return a non-zero status and provide recovery instructions.

Actual Behavior

A copy exception is printed and ignored. The command still attempts to delete the source directory and, if deletion succeeds, creates the symbolic link. Copy and finalization failures are not reliably propagated to the caller, so the command may still print move db done. and return 0.

Frequency

• Always (100%)
• Frequently (>50%)
• Sometimes (10-50%)
• Rarely (<10%)

The data-loss path is consistently reached when a file copy fails but the subsequent source deletion and symbolic-link creation succeed.

Steps to Reproduce

Use only a disposable database from a stopped node and an isolated test filesystem. Do not fill a production disk to reproduce this issue.

1. Configure the destination on an isolated test filesystem whose available space is smaller than the source database.
2. Ensure that the destination database path does not exist.
3. Run:

java -jar Toolkit.jar db mv -d output-directory -c config.conf
echo $?

4. Wait for the copy to fail because the destination filesystem runs out of space, then inspect the source path, destination directory, and exit status.

Based on the current code flow, if the subsequent deletion and symbolic-link creation succeed, the source path will become a symbolic link to an incomplete destination, while the command still reports success and returns 0.

Logs and Error Messages

The following is the expected error form for a full destination filesystem. It illustrates the current code flow and is not a captured reproduction log. Exact text depends on the operating system and filesystem.

java.nio.file.FileSystemException: <destination file>: No space left on device
move db done.
(exit code 0)

Additional Context (Optional)

Related Issues

N/A

Possible Solution

Use a two-phase migration:

1. Copy all databases first. If any copy fails, preserve every source database, attempt to remove incomplete destination data, and return a non-zero status.
2. Delete the source directories and create symbolic links only after all copies succeed.

A source deletion or symbolic-link creation failure should also return a non-zero status and provide recovery instructions.