[Task]: Move to a single classpath

### What needs to happen?

# Investigation: Migrating Apache Hop to a Single Unified Classpath

This document outlines the design, changes, and benefits of migrating Apache Hop from separate classpaths (and classloaders) per plugin to a single, unified classpath.

---

## 1. Executive Summary

Currently, Apache Hop uses a modular classloading architecture where each plugin directory (under `plugins/`) has its own dependencies and is loaded using a separate instance of `HopURLClassLoader`.

While this provides dependency isolation between plugins, it introduces:
* High classloader overhead and potential memory leaks (metaspace exhaustion).
* Complexity in containerized environments (Docker) and distributed executors (Apache Spark, Apache Flink, Google Cloud Dataflow via the Apache Beam engine), where classloading isolation causes issues.
* Difficulty in native compilation (GraalVM).

### Proposed Solution: Manifest-only Pathing Jar
To move to a single classpath, we can pre-calculate the list of all jar files in the distribution (including core, beam, and all plugins) during the maven assembly build of `hop-client` and package them into a manifest-only jar: `lib/hop-classpath.jar`.

This jar contains only a `META-INF/MANIFEST.MF` file listing all individual jar relative paths under `lib/` and `plugins/` in its `Class-Path` header. This avoids OS-specific command-line length limits (particularly the 8,191-character limit on Windows) while achieving a single classpath.

---

## 2. Java Runtime Changes

To support running on a single classpath, we can introduce a system property/environment flag, e.g., `HOP_SINGLE_CLASSPATH=Y`. When set, the `PluginRegistry` bypasses separate classloader creation and delegates all plugin loading to the system (App) classloader.

### Proposed Diff: `PluginRegistry.java`

```diff
diff --git a/core/src/main/java/org/apache/hop/core/plugins/PluginRegistry.java b/core/src/main/java/org/apache/hop/core/plugins/PluginRegistry.java
--- a/core/src/main/java/org/apache/hop/core/plugins/PluginRegistry.java
+++ b/core/src/main/java/org/apache/hop/core/plugins/PluginRegistry.java
@@ -867,6 +867,10 @@
public ClassLoader getClassLoader(IPlugin plugin) throws HopPluginException {

if (plugin == null) {
throw new HopPluginException(
BaseMessages.getString(
PKG, "PluginRegistry.RuntimeError.NoValidTransformOrPlugin.PLUGINREGISTRY001"));
}

+ // If single classpath mode is active, use the main system classloader for all plugins
+ if (plugin.isNativePlugin() || "Y".equalsIgnoreCase(System.getProperty("HOP_SINGLE_CLASSPATH"))) {
+ return this.getClass().getClassLoader();
+ }
+
try {
```

### Jandex Plugin Scanning
Because plugin jars are on the system classpath, `JarCache.getNativeJars()` will automatically discover all plugin `META-INF/jandex.idx` files and register them. The system property `HOP_SINGLE_CLASSPATH=Y` ensures that even when standard folders are scanned, they map back to the system classloader at runtime.

---

## 3. Maven Build & Pre-Calculation Changes

To pre-calculate the list of jars during the client build process, we modify the build cycle of `assemblies/client`.

### Workflow
1. **Unpack Dependencies**: Use `maven-dependency-plugin` to unpack all dependency zips (static, plugins, core, engine, etc.) into a staging directory `${project.build.directory}/stage/hop`.
2. **Pre-calculate Classpath**: Execute a Groovy script using `groovy-maven-plugin` (or `gmavenplus-plugin`) during the `prepare-package` phase to scan `${project.build.directory}/stage/hop` for all jars, compute their relative paths (excluding platform-specific SWT jars), and write a manifest-only jar `lib/hop-classpath.jar`.
3. **Assemble Zip**: Configure `maven-assembly-plugin` to pack files directly from the stage folder.

### Proposed Configuration in `assemblies/client/pom.xml`

```xml

org.apache.maven.plugins
maven-dependency-plugin


unpack-dependencies
prepare-package

unpack-dependencies


zip
${project.build.directory}/stage/hop
true

org.codehaus.gmavenplus
gmavenplus-plugin
3.0.2


prepare-package

execute



<![CDATA[
import java.util.jar.JarOutputStream
import java.util.jar.Manifest
import java.util.jar.Attributes

File stageDir = new File(project.build.directory, "stage/hop")
List<String> relativePaths = []

// Scan staging directory recursively for jar files
stageDir.eachFileRecurse { file ->
if (file.name.endsWith(".jar") && !file.path.contains("/swt/")) {
// Compute path relative to lib/ directory
String relPath = stageDir.toPath().relativize(file.toPath()).toString().replace('\\', '/')
if (relPath.startsWith("lib/")) {
relativePaths.add(relPath.substring(4))
} else {
relativePaths.add("../" + relPath)
}
}
}

// Build the manifest with the Class-Path attribute
Manifest manifest = new Manifest()
manifest.mainAttributes.put(Attributes.Name.MANIFEST_VERSION, "1.0")
manifest.mainAttributes.put(Attributes.Name.CLASS_PATH, relativePaths.join(" "))

// Write manifest-only pathing jar
File classpathJar = new File(stageDir, "lib/hop-classpath.jar")
classpathJar.parentFile.mkdirs()
new JarOutputStream(new FileOutputStream(classpathJar), manifest).close()
log.info("Generated pre-calculated classpath manifest jar: " + classpathJar)
]]>

```

### Proposed Changes to `assemblies/client/src/assembly/assembly.xml`

```xml



${project.build.directory}/stage/hop
.


```

---

## 4. Startup Script Changes

In the shell and batch scripts, we replace the directory wildcards (`lib/core/*:lib/beam/*`) with `lib/hop-classpath.jar`, dynamically adding only the platform-dependent SWT library jar. We also pass the `-DHOP_SINGLE_CLASSPATH=Y` runtime option.

### A. Linux & OSX Scripts (`.sh` scripts: `hop-gui.sh`, `hop-run.sh`, etc.)

```diff
- if "${_HOP_JAVA}" -XshowSettings:properties -version 2>&1 | grep -q "os.arch = aarch64"; then
- CLASSPATH="lib/core/*:lib/beam/*:lib/swt/linux/arm64/*"
- else
- CLASSPATH="lib/core/*:lib/beam/*:lib/swt/linux/$(uname -m)/*"
- fi
+ if "${_HOP_JAVA}" -XshowSettings:properties -version 2>&1 | grep -q "os.arch = aarch64"; then
+ SWT_JAR="lib/swt/linux/arm64/swt.jar"
+ else
+ SWT_JAR="lib/swt/linux/$(uname -m)/swt.jar"
+ fi
+ CLASSPATH="lib/hop-classpath.jar:${SWT_JAR}"
+ HOP_OPTIONS="${HOP_OPTIONS} -DHOP_SINGLE_CLASSPATH=Y"
```

### B. Windows Scripts (`.bat` scripts: `hop-gui.bat`, `hop-run.bat`, etc.)

```diff
-set CLASSPATH=lib\core\*;lib\beam\*;lib\swt\win64\*
+set CLASSPATH=lib\hop-classpath.jar;lib\swt\win64\swt.jar
+set HOP_OPTIONS=%HOP_OPTIONS% -DHOP_SINGLE_CLASSPATH=Y
```

---

## 5. Benefits & Drawbacks

### Benefits
* **Significantly Simplified Classloading**: Standard class resolution is handled directly by the JVM system classloader.
* **Massive Reduction in Overhead**: Avoids spawning hundreds of `HopURLClassLoader` instances, reducing JVM Metaspace consumption and startup time.
* **Distributed Compatibility**: Facilitates executing pipelines on external clusters (Spark/Flink) since the entire Hop execution engine + plugins can be shipped using regular classpaths/jars.
* **Solves Windows Command Limit**: The manifest jar keeps the execution command short, preventing Windows 8,191-character limit crashes.

### Drawbacks & Mitigation
* **Dependency Conflicts**: If two plugins include different versions of the same third-party library, classpath conflicts (NoSuchMethodError, etc.) may arise.
* *Mitigation*: These conflicts are now surfaced at compile/build time when constructing the client rather than silently failing or causing classloader issues at runtime. Surfacing this enforces dependency alignment across the project.
* **Dynamic Loading of JDBC Drivers**: The driver installer hot-loading logic (`DriverInstaller.hotLoad()`) expects `HopURLClassLoader` to inject downloaded JDBC drivers.
* *Mitigation*: Users should add custom JDBC drivers to the shared folders (`HOP_SHARED_JDBC_FOLDERS`), or place them directly in the classpath via options.

### Issue Priority

Priority: 3

### Issue Component

Component: API

Start by reading core/src/main/java/org/apache/hop/core/plugins/PluginRegistry.java and the current build and assembly configuration in assemblies/client/pom.xml and assemblies/client/src/assembly/assembly.xml. Then inspect the hop-gui.sh, hop-run.sh, and corresponding .bat startup scripts. Done means the client packages a usable hop-classpath.jar and the startup paths support single-classpath mode across the documented platforms.

Written by the indexing model from the issue text.