Self-Hosting Open-Source AI Models: The Hardware You Need Before You Start

Running a language model on your own machine is no longer exotic — the software is a two-line install and the models are free to download. What stops most attempts is hardware, and specifically memory. The good news is that the question “can my machine run this?” has a fairly precise answer, and it is decided before you download anything.

This guide covers what determines whether a model fits, what each hardware tier can realistically run, and when renting capacity is simply the better decision.

What decides whether a model fits your machine

Three factors decide it. Model size, measured in billions of parameters, sets the baseline memory requirement. Quantisation, which compresses the weights, can cut that requirement by more than half. Context length — how much text the model holds at once — adds memory on top, and grows with every conversation you keep open.

If the model and its context do not fit in memory, the system falls back to using the disk, and inference slows to the point of being unusable. That is the failure everybody hits first, and it is why the memory figure matters more than the processor.

How memory, not compute, sets the ceiling

A useful rule of thumb: a seven-billion-parameter model in four-bit quantisation needs roughly four to five gigabytes of memory for the weights alone, before any context. A thirteen-billion model at the same quantisation lands near eight gigabytes, and a thirty-billion model near eighteen. Add a couple of gigabytes for context, the operating system and your browser, and the practical requirement is usually fifty per cent above the weights.

Compute then decides speed rather than feasibility. A dedicated GPU generates tokens several times faster than a CPU, which is why the same model feels instant on one machine and painful on another with identical memory.

What each hardware tier can actually run

Hardware	What runs comfortably	Realistic experience
Laptop, 16 GB unified memory	Small 3-8B models, quantised	Fine for drafts and summaries; slow for long context
Desktop with 12-16 GB VRAM	8-14B models	Interactive; good for coding assistance
24 GB VRAM	Up to roughly 30B quantised	Comfortable for most local work
Multiple GPUs or 64 GB+ unified	70B class, quantised	Serious workloads, serious power draw and heat
CPU only, 32 GB RAM	Small models	Works, but measured in seconds per response

Why quantisation changes the answer

Quantisation stores each weight with fewer bits. At four bits instead of sixteen, the same model occupies about a quarter of the memory, with a quality loss that is noticeable mainly on reasoning-heavy tasks. For summarising, drafting, classification and code completion, the difference is usually acceptable; for anything that needs precise multi-step reasoning, it is worth keeping the higher precision and choosing a smaller model instead.

The practical consequence is that a mid-range machine can run a useful model today, as long as the expectation is adjusted: smaller and quantised, rather than the largest model available.

What to prepare before downloading a model

Check four things first. Free disk space: model files are large, and keeping two or three costs more than people expect. Power and cooling: sustained inference turns a quiet desktop into a warm one, and a laptop on battery will throttle. Software: a local runner such as Ollama or LM Studio handles model management and exposes an HTTP endpoint, which is what lets your own code call it. Network: downloading twenty gigabytes over a slow connection is its own afternoon.

When renting a GPU beats owning one

Ownership wins when usage is constant and the data cannot leave your premises. Renting wins when usage is occasional, because the cost of a capable GPU, the electricity and the depreciation of a machine that is idle most of the week rarely beat an hourly rental. A hybrid pattern is common and sensible: a small local model for drafts, summaries and anything sensitive, and a paid API for the heavy requests.

For the specifics, the list of open-source models worth self-hosting covers what is available and what each one needs, the free AI API roundup covers the alternative when local hardware is not enough, and wrapping a model in a FastAPI service is the quickest way to put a local model behind an endpoint your apps can call.