microsoft / microsoft/foundry-local

[Feature Request] Reduce disk duplication across CPU/GPU/NPU model variants by sharing common model weights

Open
#1,040 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

### Describe the feature request

## Describe the feature request*

Foundry Local currently provides separate hardware-specific model variants for CPU, GPU, and NPU execution.

For the same logical model, users may need to download and cache multiple variants if they want to switch between CPU, GPU, and NPU. For large models, this can result in significant duplicated storage usage.

I would like Foundry Local to support a shared model storage architecture where common model data, especially large model weights and tokenizer files, can be reused across CPU/GPU/NPU variants whenever technically possible.

For example:

```text id="ejv639"
Model
├── shared/
│ ├── weights
│ └── tokenizer
├── cpu/
│ └── device-specific artifacts
├── gpu/
│ └── device-specific artifacts
└── npu/
└── device-specific artifacts
```

The request is not to remove hardware-specific variants completely. I understand that different devices may require different quantization formats, graph optimizations, execution-provider-specific layouts, or compiled artifacts.

Instead, Foundry Local could deduplicate or share identical portions of model variants, while storing only genuinely hardware-specific data separately.

Possible implementations could include shared base weights, content-addressable storage, block-level deduplication, or runtime-generated per-device optimization caches.

### Describe scenario use case

## Describe scenario use case*

A typical AI PC may contain all three compute devices:

```text id="1gbo3q"
CPU
GPU
NPU
```

As a developer, I may want to allow users to select the inference device dynamically:

```text id="pwd771"
Inference Device

○ Auto
○ CPU
○ GPU
○ NPU
```

This is useful for several scenarios:

* Benchmarking CPU, GPU, and NPU performance on the same machine.
* Comparing power consumption and performance between devices.
* Allowing users to choose between performance and power efficiency.
* Falling back to CPU when GPU or NPU resources are unavailable.
* Testing compatibility across different execution providers.
* Developing applications that can dynamically select the most appropriate compute device.

With the current hardware-variant model, supporting this scenario may require downloading three separate variants of the same model.

For small models this may be acceptable, but for larger models the storage overhead becomes significant.

For example, if each hardware variant is approximately 10 GB:

```text id="vxwc9s"
CPU variant ~10 GB
GPU variant ~10 GB
NPU variant ~10 GB
----------------------
Total ~30 GB
```

Ideally, the large common model data would only need to be stored once, with smaller hardware-specific artifacts cached separately.

### Customer requirement and impact

## Customer requirement and impact*

The main requirement is to reduce duplicated disk usage when the same logical model is installed for multiple hardware targets.

This becomes increasingly important as model sizes grow.

Users with CPU, GPU, and NPU capable systems should ideally be able to take advantage of all available compute devices without having to maintain multiple full copies of the same model whenever a significant portion of the underlying data is identical.

The expected benefits would include:

* Reduced disk usage for multi-device model installations.
* Reduced download size when adding another hardware variant.
* More practical CPU/GPU/NPU switching.
* Better support for large local models.
* Easier benchmarking and development across different accelerators.
* Better utilization of modern AI PCs containing multiple compute devices.
* More efficient model cache management.

For example, a large model may require tens of gigabytes per variant. Keeping CPU, GPU, and NPU versions could therefore consume tens or even hundreds of gigabytes of additional storage.

Even partial deduplication of shared model weights would significantly improve this situation.

A preferred design would be:

```text id="nk4lmt"
Shared model weights

├── CPU-specific cache/artifacts
├── GPU-specific cache/artifacts
└── NPU-specific cache/artifacts
```

rather than requiring three fully independent model copies whenever the underlying weights can be shared.

This would preserve Foundry Local's hardware-optimized variant architecture while making multi-device usage much more storage-efficient.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are identified in the request, so begin by locating Foundry Local’s model download, cache, and CPU/GPU/NPU variant handling. Done would require an agreed design and implementation that reuses common model data while preserving device-specific artifacts, with tests demonstrating reduced duplication and correct switching.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.