dotnet / dotnet/eShopSupport

Add instructions for using this app on a system with no GPU

Open
#19 2 comments 7 reactions 1 assignee Claimed by @jmatthiesen View on GitHub
Dominant language
JavaScript
Stars
658
Forks
189
PR merge metrics
No merged PRs in 30d

Description

This sample will work best on a system that has a GPU, but it can be used on a system without one if necessary. As we work on setup instructions for this app, we should include details for getting it to work on these systems with no GPU. The following are the steps I had to take so far to get it working and I'll keep adding to this until I can use all the sample use cases:

1. Update the Ollama bootstrapping code so it does not use a GPU. In the AppHost project, Program.cs change this code:

```csharp
var chatCompletion = builder.AddOllama("chatcompletion").WithDataVolume();
```

to

```csharp
var chatCompletion = builder.AddOllama("chatcompletion", enableGpu: false).WithDataVolume();
```

1. In the PythonInference project, change requirements.txt to use versions of Torch libraries without CUDA. Change it from:

```
--extra-index-url https://download.pytorch.org/whl/cu118
torch==2.3.1+cu118
torchaudio==2.3.1+cu118
torchvision==0.18.1+cu118
```

to:

```
torch==2.3.1
torchaudio==2.3.1
torchvision==0.18.1
```

1. Change the PythonInference project so that it does not use CUDA for accessing models. In the routers/classifier.py file, change this line from:

```python
classifier = pipeline('zero-shot-classification', model='cross-encoder/nli-MiniLM2-L6-H768', device='cuda')
```

to:

```python
classifier = pipeline('zero-shot-classification', model='cross-encoder/nli-MiniLM2-L6-H768')
```

1. When running without a GPU, responses from the models will be slower and default settings for timeouts are not enough. I had to update my ServiceDefaults project, Extensions.cs file to increase the StandardResilience timeouts. The following worked for me, but on some systems a different timeout may be needed. In Extensions.AddServiceDefaults, change the StandardResilienceHandler from:

```csharp
http.AddStandardResilienceHandler();
```

to:

```csharp
http.AddStandardResilienceHandler(options =>
{
options.AttemptTimeout = new HttpTimeoutStrategyOptions
{
Timeout = TimeSpan.FromMinutes(10)
};
options.TotalRequestTimeout = new HttpTimeoutStrategyOptions
{
Timeout = TimeSpan.FromMinutes(10)
};
options.CircuitBreaker.SamplingDuration = TimeSpan.FromMinutes(20);
});
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.