How to Run Large Language Models Locally with Ollama: Complete Setup Guide

Want to run powerful AI models like Llama 3, Mistral, or Gemma directly on your own machine—without cloud subscriptions or API fees? The answer is Ollama, a free, open-source tool that lets you download, run, and manage large language models (LLMs) locally in minutes. This guide walks you through the complete setup, from installation to advanced Docker deployment, giving you full privacy, offline access, and unlimited usage.

Ollama terminal interface showing model download progress

Photo by Brett Sayles on Pexels

## Why Run LLMs Locally?

Running models locally puts you in control. Here’s why it matters:

  • Complete privacy – No data leaves your machine
  • Zero ongoing costs – No API keys or monthly subscriptions
  • Offline access – Use AI without internet
  • Customization – Fine-tune and experiment freely
  • Low latency – Instant responses without network delays

## What You Need Before Starting

### Hardware Requirements

  • CPU: 4+ cores (Intel/AMD modern processor)
  • RAM: 8GB minimum (16GB+ recommended for 7B models)
  • GPU (optional): NVIDIA with 6GB+ VRAM for acceleration
  • Storage: 5-50GB free space per model

### Software Prerequisites

  • Windows: Windows 10/11, or WSL2 (recommended)
  • macOS: macOS 12+ (Monterey or later)
  • Linux: Ubuntu 20.04+, Fedora 36+, or Debian 11+
  • Docker (for container deployment)

## Step 1: Install Ollama

### On Windows

1. Download the installer from ollama.com

2. Run the `.exe` file and follow setup wizard

3. Open Command Prompt or PowerShell

4. Verify installation: `ollama –version`

### On macOS

“`bash

# Using Homebrew (recommended)

brew install ollama

# Or download DMG from ollama.com

“`

### On Linux

“`bash

curl -fsSL https://ollama.com/install.sh | sh

“`

## Step 2: Download and Run Your First Model

Ollama supports dozens of models. Start with a lightweight one:

“`bash

# Download and run Llama 3.2 (3B parameters)

ollama run llama3.2

# Or try Mistral (7B, great all-rounder)

ollama run mistral

# For coding tasks

ollama run codellama

“`

Pro tip: Begin with 3B models for testing. Upgrade to 7B or 13B once comfortable.

Ollama chat interface with user typing a question and getting AI response

Photo by Matheus Bertelli on Pexels

## Step 3: Basic Commands for Daily Use

Command Description
`ollama list` Show downloaded models
`ollama pull modelname` Download without running
`ollama rm modelname` Delete a model
`ollama stop` Stop current model
`ollama serve` Start API server

### Interactive vs. One-shot Mode

Interactive mode (default):

“`

ollama run llama3.2

>>> Write a haiku about AI

“`

One-shot mode (for scripts):

“`bash

ollama run llama3.2 “Explain quantum computing in 50 words”

“`

## Step 4: Docker Deployment for Production

Deploying Ollama in Docker gives you portability, isolation, and easier scaling.

### Basic Docker Setup

“`bash

# Pull the official image

docker pull ollama/ollama

# Run with CPU

docker run -d -v ollama:/root/.ollama -p 11434:11434 –name ollama ollama/ollama

# Run with GPU (NVIDIA)

docker run -d –gpus all -v ollama:/root/.ollama -p 11434:11434 –name ollama ollama/ollama

“`

### Docker Compose for Full Stack

Create a `docker-compose.yml` file:

“`yaml

version: ‘3.8’

services:

ollama:

image: ollama/ollama:latest

container_name: ollama

ports:

  • “11434:11434”

volumes:

  • ollama_data:/root/.ollama

deploy:

resources:

reservations:

devices:

  • capabilities: [gpu]

restart: unless-stopped

open-webui:

image: ghcr.io/open-webui/open-webui:main

container_name: open-webui

ports:

  • “3000:8080”

volumes:

  • open-webui_data:/app/backend/data

environment:

  • OLLAMA_BASE_URL=http://ollama:11434

depends_on:

  • ollama

restart: unless-stopped

volumes:

ollama_data:

open-webui_data:

“`

Then run:

“`bash

docker compose up -d

“`

Access the web interface at `http://localhost:3000`

Docker terminal showing running containers with Ollama and Open WebUI

Photo by Wolfgang Weiser on Pexels

## Step 5: Optimize Performance

### For CPU-Only Systems

  • Use quantized models (Q4_K_M suffix) – they use 4x less memory
  • Limit context window: `ollama run mistral –num-ctx 2048`
  • Close other heavy applications

### For GPU Acceleration

  • Install NVIDIA Container Toolkit for Docker
  • Check GPU memory: `nvidia-smi`
  • Use models optimized for your GPU (e.g., `llama3.2:7b-instruct-q4_K_M`)

### Memory Management

“`bash

# Set maximum context length (reduces RAM usage)

ollama run llama3.2 –num-ctx 4096

# Use half-precision (faster on compatible GPUs)

OLLAMA_KEEP_ALIVE=0 ollama run mistral

“`

## Advanced Tips for Power Users

### 1. Create Custom Models with Modelfiles

“`dockerfile

FROM llama3.2

# Set custom temperature

PARAMETER temperature 0.7

# Add system prompt

SYSTEM “You are a helpful coding assistant. Always provide examples.”

“`

Build it:

“`bash

ollama create mycoder -f ./Modelfile

“`

### 2. Use the REST API

Ollama exposes a REST API on port 11434:

“`bash

curl http://localhost:11434/api/generate -d ‘{

“model”: “llama3.2”,

“prompt”: “What is Docker?”,

“stream”: false

}’

“`

### 3. Automate with Scripts

“`bash

#!/bin/bash

# Batch process text files

for file in *.txt; do

ollama run mistral “Summarize this: $(cat $file)” > “${file%.txt}_summary.txt”

done

“`

## Troubleshooting Common Issues

Problem Solution
“Out of memory” Use smaller model or quantized version
Slow response Enable GPU acceleration or reduce context
Model not found Check name spelling, try `ollama pull`
Docker port conflict Change port mapping in docker-compose
CUDA errors Update NVIDIA drivers and container toolkit

## What Models Should You Try?

Here’s a quick comparison:

Model Size Best For RAM Needed
Llama 3.2 (3B) 2.0GB General chat, quick tasks 4GB
Mistral (7B) 4.1GB All-purpose, reasoning 8GB
CodeLlama (7B) 3.8GB Programming, debugging 8GB
Gemma 2 (9B) 5.5GB Creative writing, analysis 12GB
Llama 3.1 (8B) 4.7GB Complex tasks, instruction 10GB

## Conclusion

Running LLMs locally with Ollama puts cutting-edge AI at your fingertips—privately, freely, and offline. Whether you’re a developer deploying with Docker, a writer exploring creative tools, or a privacy-conscious user, Ollama simplifies the process to just a few commands.

Start with a small model, experiment with the API, and scale up as you learn. The local AI revolution is here, and you’re already part of it.

Ready to take your AI setup further? Visit freeresource.store for exclusive Docker templates, pre-configured Modelfiles, and open-source deployment scripts that save hours of setup time. Join our community of AI enthusiasts and get access to premium resources—completely free.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *