bmaltais / bmaltais/orchestrator

MVP-3: Optimize Ollama configuration on RTX 3090

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Problem
Ollama running on Windows machine with RTX 3090 may not be optimally configured:
- Model loading parameters may not leverage GPU fully
- Quantization choices may impact performance
- Concurrency settings may not be optimal
- Memory allocation may be inefficient

## Current State
- Ollama service: `http://ollama-desktop:11434`
- Models: qwen3.5:latest, nomic-embed-text, llava:13b-v1.6
- Hardware: RTX 3090 (24GB VRAM)

## Requirements
1. Assess current Ollama configuration
2. Optimize GPU layer allocation
3. Tune concurrency settings
4. Evaluate quantization levels
5. Configure memory management
6. Add performance monitoring

## Implementation Plan
1. Check current Ollama configuration
2. Review model loading parameters
3. Optimize GPU utilization settings
4. Test performance with different configurations
5. Document optimal settings

## Acceptance Criteria
- GPU is fully utilized for model inference
- Response times are optimized
- Memory usage is efficient
- Concurrent requests are handled properly

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting the Ollama service at http://ollama-desktop:11434 and recording configuration, model-loading behavior, and baseline GPU, memory, latency, and concurrency metrics for qwen3.5:latest, nomic-embed-text, and llava:13b-v1.6. Compare GPU allocation, concurrency, quantization, and memory settings on the RTX 3090, then document the tested configuration and results that satisfy the acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama
Domain
ai-infra-agents, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.