How to Run Large Language Models Locally with Ollama: Complete Setup Guide
Want to run powerful AI models like Llama 3, Mistral, or Gemma directly on your own machine—without cloud subscriptions or API fees? The answer is Ollama, a free, open-source tool that lets you download, run, and manage large language models (LLMs) locally in minutes. This guide walks you through the complete setup, from installation to advanced Docker deployment, giving you full privacy, offline access, and unlimited usage.

Photo by Brett Sayles on Pexels
## Why Run LLMs Locally?
Running models locally puts you in control. Here’s why it matters:
- Complete privacy – No data leaves your machine
- Zero ongoing costs – No API keys or monthly subscriptions
- Offline access – Use AI without internet
- Customization – Fine-tune and experiment freely
- Low latency – Instant responses without network delays
## What You Need Before Starting
### Hardware Requirements
- CPU: 4+ cores (Intel/AMD modern processor)
- RAM: 8GB minimum (16GB+ recommended for 7B models)
- GPU (optional): NVIDIA with 6GB+ VRAM for acceleration
- Storage: 5-50GB free space per model
### Software Prerequisites
- Windows: Windows 10/11, or WSL2 (recommended)
- macOS: macOS 12+ (Monterey or later)
- Linux: Ubuntu 20.04+, Fedora 36+, or Debian 11+
- Docker (for container deployment)
## Step 1: Install Ollama
### On Windows
1. Download the installer from ollama.com
2. Run the `.exe` file and follow setup wizard
3. Open Command Prompt or PowerShell
4. Verify installation: `ollama –version`
### On macOS
“`bash
# Using Homebrew (recommended)
brew install ollama
# Or download DMG from ollama.com
“`
### On Linux
“`bash
curl -fsSL https://ollama.com/install.sh | sh
“`
## Step 2: Download and Run Your First Model
Ollama supports dozens of models. Start with a lightweight one:
“`bash
# Download and run Llama 3.2 (3B parameters)
ollama run llama3.2
# Or try Mistral (7B, great all-rounder)
ollama run mistral
# For coding tasks
ollama run codellama
“`
Pro tip: Begin with 3B models for testing. Upgrade to 7B or 13B once comfortable.

Photo by Matheus Bertelli on Pexels
## Step 3: Basic Commands for Daily Use
| Command | Description |
|---|---|
| `ollama list` | Show downloaded models |
| `ollama pull modelname` | Download without running |
| `ollama rm modelname` | Delete a model |
| `ollama stop` | Stop current model |
| `ollama serve` | Start API server |
### Interactive vs. One-shot Mode
Interactive mode (default):
“`
ollama run llama3.2
>>> Write a haiku about AI
“`
One-shot mode (for scripts):
“`bash
ollama run llama3.2 “Explain quantum computing in 50 words”
“`
## Step 4: Docker Deployment for Production
Deploying Ollama in Docker gives you portability, isolation, and easier scaling.
### Basic Docker Setup
“`bash
# Pull the official image
docker pull ollama/ollama
# Run with CPU
docker run -d -v ollama:/root/.ollama -p 11434:11434 –name ollama ollama/ollama
# Run with GPU (NVIDIA)
docker run -d –gpus all -v ollama:/root/.ollama -p 11434:11434 –name ollama ollama/ollama
“`
### Docker Compose for Full Stack
Create a `docker-compose.yml` file:
“`yaml
version: ‘3.8’
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
ports:
- “11434:11434”
volumes:
- ollama_data:/root/.ollama
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]
restart: unless-stopped
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
ports:
- “3000:8080”
volumes:
- open-webui_data:/app/backend/data
environment:
- OLLAMA_BASE_URL=http://ollama:11434
depends_on:
- ollama
restart: unless-stopped
volumes:
ollama_data:
open-webui_data:
“`
Then run:
“`bash
docker compose up -d
“`
Access the web interface at `http://localhost:3000`

Photo by Wolfgang Weiser on Pexels
## Step 5: Optimize Performance
### For CPU-Only Systems
- Use quantized models (Q4_K_M suffix) – they use 4x less memory
- Limit context window: `ollama run mistral –num-ctx 2048`
- Close other heavy applications
### For GPU Acceleration
- Install NVIDIA Container Toolkit for Docker
- Check GPU memory: `nvidia-smi`
- Use models optimized for your GPU (e.g., `llama3.2:7b-instruct-q4_K_M`)
### Memory Management
“`bash
# Set maximum context length (reduces RAM usage)
ollama run llama3.2 –num-ctx 4096
# Use half-precision (faster on compatible GPUs)
OLLAMA_KEEP_ALIVE=0 ollama run mistral
“`
## Advanced Tips for Power Users
### 1. Create Custom Models with Modelfiles
“`dockerfile
FROM llama3.2
# Set custom temperature
PARAMETER temperature 0.7
# Add system prompt
SYSTEM “You are a helpful coding assistant. Always provide examples.”
“`
Build it:
“`bash
ollama create mycoder -f ./Modelfile
“`
### 2. Use the REST API
Ollama exposes a REST API on port 11434:
“`bash
curl http://localhost:11434/api/generate -d ‘{
“model”: “llama3.2”,
“prompt”: “What is Docker?”,
“stream”: false
}’
“`
### 3. Automate with Scripts
“`bash
#!/bin/bash
# Batch process text files
for file in *.txt; do
ollama run mistral “Summarize this: $(cat $file)” > “${file%.txt}_summary.txt”
done
“`
## Troubleshooting Common Issues
| Problem | Solution |
|---|---|
| “Out of memory” | Use smaller model or quantized version |
| Slow response | Enable GPU acceleration or reduce context |
| Model not found | Check name spelling, try `ollama pull` |
| Docker port conflict | Change port mapping in docker-compose |
| CUDA errors | Update NVIDIA drivers and container toolkit |
## What Models Should You Try?
Here’s a quick comparison:
| Model | Size | Best For | RAM Needed |
|---|---|---|---|
| Llama 3.2 (3B) | 2.0GB | General chat, quick tasks | 4GB |
| Mistral (7B) | 4.1GB | All-purpose, reasoning | 8GB |
| CodeLlama (7B) | 3.8GB | Programming, debugging | 8GB |
| Gemma 2 (9B) | 5.5GB | Creative writing, analysis | 12GB |
| Llama 3.1 (8B) | 4.7GB | Complex tasks, instruction | 10GB |
## Conclusion
Running LLMs locally with Ollama puts cutting-edge AI at your fingertips—privately, freely, and offline. Whether you’re a developer deploying with Docker, a writer exploring creative tools, or a privacy-conscious user, Ollama simplifies the process to just a few commands.
Start with a small model, experiment with the API, and scale up as you learn. The local AI revolution is here, and you’re already part of it.
Ready to take your AI setup further? Visit freeresource.store for exclusive Docker templates, pre-configured Modelfiles, and open-source deployment scripts that save hours of setup time. Join our community of AI enthusiasts and get access to premium resources—completely free.