[SIP-159] Proposal for Migrating from npm + Lerna to pnpm

## [SIP] Proposal for Migrating from npm + Lerna to pnpm

### Motivation
Apache Superset’s frontend currently relies on **npm** for dependency management and **Lerna** for monorepo publishing and workspace organization. While Lerna has served us well, it has experienced periods of limited maintenance and slower updates. Additionally, we still face challenges with large `node_modules` directories, long install times, and complex multi-package workflows.

**pnpm** offers a compelling alternative that consolidates both package management and monorepo workspace features in a single tool. Some key benefits include:

- **Smaller `node_modules` footprint**: pnpm’s content-addressable store greatly reduces disk usage.
- **Faster installations**: pnpm’s symlink-based architecture speeds up the dependency installation process.
- **Actively maintained**: pnpm is supported by the npm team, ensuring ongoing development and timely fixes.
- **Built-in workspace management**: pnpm can replace Lerna’s monorepo features without additional overhead.

### Description of the problem to be solved
1. **Large `node_modules` directories**: Both local development environments and CI pipelines suffer from bloated install footprints.
2. **Long install times**: As dependencies grow, `npm install` becomes slower and more resource-intensive.
3. **Maintenance overhead**: Using two tools—npm and Lerna—for monorepo management can lead to redundant configuration and potential version mismatches.
4. **Future-proofing**: Lerna’s maintenance status has fluctuated, creating uncertainty for the project’s long-term needs.

### Proposed Change
1. **Adopt pnpm Workspaces**
- Migrate from Lerna’s monorepo setup (`lerna.json` and associated scripts) to a **`pnpm-workspace.yaml`** configuration.
- Remove or deprecate Lerna-specific commands in favor of pnpm’s built-in workspace features.
2. **Replace npm with pnpm**
- Update all `npm install` and `npm run` scripts to use **pnpm**.
- Validate and adjust any scripts or hooks to ensure they function under pnpm.
3. **Integrate Changesets for Versioning & Publishing** (optional but recommended)
- If we still want automated changelog generation and version bumping, incorporate [Changesets](https://github.com/changesets/changesets).
- Configure pnpm to run Changesets during CI to publish packages.
4. **Update CI/CD**
- Switch CI steps from `npm install` to `pnpm install`.
- Evaluate caching strategies (e.g., caching the pnpm store) to maximize build performance.
5. **Documentation**
- Provide clear migration steps for developers (e.g., uninstall Lerna globally if used, install pnpm, and switch to `pnpm` commands).
- Update any references to Lerna or npm in project documentation, READMEs, and onboarding guides.

### New or Changed Public Interfaces
- **No direct impact on Superset’s REST endpoints, dashboards, or CLI** is anticipated.
- Developer-facing scripts (e.g., `lerna publish` or `npm run build`) will be replaced by `pnpm run publish` or `pnpm run build`, necessitating documentation updates.

### New Dependencies
- **pnpm**
- License: MIT
- Actively maintained by the npm team and open-source community.
- **(Optional) Changesets**
- License: MIT
- Actively maintained and widely adopted for multi-package versioning and changelog generation.

### Migration Plan and Compatibility
1. **Local Environment**
- Developers will install pnpm globally (`corepack enable pnpm` or others).
- Replace Lerna commands (e.g., `lerna bootstrap`, `lerna publish`) with pnpm equivalents (`pnpm install`, `pnpm publish`).
2. **CI Environments**
- Update pipelines to install pnpm and run `pnpm install` instead of `npm install` or `lerna bootstrap`.
- Validate build scripts, tests, and publishing flows under pnpm.
3. **Backward Compatibility**
- Removing Lerna does not affect final build artifacts or runtime usage.
- The main changes are internal to developer workflows and CI processes.

### Rejected Alternatives
1. **Continue using npm + Lerna**
- Maintains the status quo but does not address large `node_modules`, slower install times, or the uncertain maintenance status of Lerna.
2. **Nx or Turborepo**
- Both offer monorepo solutions, but either introduce additional layers of complexity or lack out-of-the-box publishing features (like Nx or Turborepo).

---

By migrating from **npm + Lerna** to **pnpm**, Apache Superset can simplify its monorepo workflows, reduce disk usage, speed up installation times, and rely on a single, well-maintained solution for both dependency management and workspace organization. This proposal aims to streamline developer workflows and position Superset for future growth and maintenance.

#32692

Start by inspecting lerna.json and the repository's CI configuration, scripts, READMEs, and onboarding documentation for npm and Lerna references. Inventory the affected install, workspace, publishing, and caching workflows before deciding on the migration details. Done means the selected workspace and versioning setup, CI flows, developer scripts, and documentation are consistently updated and validated.

Written by the indexing model from the issue text.