Run AI Locally

12:16 The easiest way to run AI on your own computer is to install Ollama or LM Studio, both free, and download an open model that fits your memory. Pick the tool: LM Studio if you want a desktop app with a model browser, Ollama if you are happy in a terminal or want a model behind other apps, llama.cpp if you want full control. Pick the model by memory: with 8 GB, a 2B to 4B model such as Gemma 4 E4B; with 16 GB, gpt-oss-20b or a 9B Qwen; with a 24 GB graphics card or a 32 GB Mac, Gemma 4 31B or Qwen3.8 27B. Nothing you type leaves the machine, and it works offline.

1:11 Local AI makes sense for privacy, offline use, cost at high volume and tinkering. It is not the way to get the best answers: the top models on the AI Leaderboards (Gemini, Claude, GPT) are closed and run only in the cloud, and the strongest open models, such as DeepSeek V4 or Mistral Large 4, need server hardware. For those, use a hosted service such as OpenRouter or Hugging Face.

The tools compared

Sources: each tool's site, read Nov 6, 2026. All three run GGUF model files, and Ollama and LM Studio also use Apple's MLX on Apple Silicon. Other options include Jan and GPT4All (desktop apps) and vLLM (for servers).

Step by step with Ollama

1. Install from ollama.com/download, or on macOS and Linux run curl -fsSL https://ollama.com/install.sh | sh.
2. Run ollama run gemma4. It downloads Google's Gemma 4 E4B (a 6.6 to 9.5 GB download) and opens a chat.
3. Try others: ollama run gpt-oss:20b, ollama run qwen3.5:9b. ollama ls lists what you have, ollama rm deletes one (Ollama CLI).
4. To use it from other apps, point them at http://localhost:11434/v1 (OpenAI compatibility); ollama launch sets up Claude Code, Codex and others in one step.

13:49 In LM Studio the same steps are clicks: search for a model, pick a size the app marks as fitting, download, chat. In an Aug 31, 2026 video, Tech With Tim loaded Gemma 4 in LM Studio and chatted with it at about 120 tokens per second on a dedicated graphics card.

How much memory do I need?

5:18 The rule: a model needs about as much memory as its download, plus room for the conversation. On a PC, what counts is the graphics card's memory (VRAM); a model that spills into ordinary RAM runs much slower. On an Apple Silicon Mac, most of the unified memory is available to the model, which is why Macs run larger models than PCs with similar budgets.

Download sizes from the Ollama library, read Nov 6, 2026. Google publishes exact memory figures for Gemma 4 at each precision: the 31B needs 17.5 GB at 4-bit and 69.9 GB at 16-bit, weights only.

3:39 Quantization is what makes this work: models are shrunk from 16-bit to 4-bit or 8-bit numbers, cutting memory by two to four times for a small loss in quality. llama.cpp supports 1.5-bit to 8-bit quantization; 4-bit (Q4) is the usual default, and the tools pick it for you.

Mixture of experts models, such as Gemma 4 26B A4B or gpt-oss, run only part of their weights for each word, so they answer faster than a dense model of the same size, but still need memory for all of it (Google).

Which local model is best?

These are starting points. Google said Gemma 4 31B ranked third among open models on Arena's text leaderboard at its Apr 2026 launch. The highest-scoring open models on the Artificial Analysis index (Kimi K3.1 Pro, DeepSeek V4.1 Flash, GLM-5.3) are server-sized; Kimi, DeepSeek and GLM have the details. Llama 4 still runs well but has had no new release since Apr 2025.

Local AI vs the cloud

Local wins on privacy (nothing leaves the machine), offline use, no per-message cost, no usage limits and freedom to change the model. The cloud wins on quality, since the leading models are closed; on speed for large models; and on features such as web search, memory and image generation that come built into ChatGPT, Claude and Gemini. A middle path: Ollama's and LM Studio's paid plans run large open models on their servers (Ollama says it keeps zero data and never trains on it), and OpenRouter reaches hundreds of models through one API. Compare policies on AI Privacy Compared.

How we chose

Tools were chosen for being free, maintained and widely supported; their facts come from their own sites and docs. Models were chosen from makers' model cards and Ollama's library (download sizes read Nov 6, 2026), across sizes and licenses, with quality signals from Arena and Artificial Analysis as cited. No benchmark here is our own.

Frequently asked questions

Can I run AI locally for free? Yes. Ollama, LM Studio and llama.cpp are free, and open models such as Gemma 4, Qwen and gpt-oss are free to download; you need only a computer with enough memory, as of Nov 6, 2026.

What is the best local LLM? For most people in Nov 2026, Gemma 4 (E4B on a laptop, 31B on a 24 GB graphics card), gpt-oss-20b on a 16 GB machine, or Qwen3.8 27B for coding are strong starting points.

How much RAM do I need to run an LLM locally? About as much free memory as the model's download plus room for the conversation: 8 GB runs 2B to 4B models, 16 GB runs gpt-oss-20b or 9B models, and 24 to 32 GB runs 27B to 31B models.

Is Ollama or LM Studio better? LM Studio is the easier desktop app with a model browser; Ollama is lighter, open source and better as a background service for other apps. Both are free and run the same models.

Can I run ChatGPT locally? Not ChatGPT itself, but OpenAI's open-weight gpt-oss-20b runs locally in 16 GB of memory, and gpt-oss-120b on an 80 GB GPU.

Can a Mac run local AI? Yes. Apple Silicon Macs run local models well because the model can use most of the unified memory; LM Studio needs an M1 or later with macOS 14, and Ollama and LM Studio both use Apple's MLX for speed.

Sources

• Ollama, GitHub, read Nov 6, 2026
• Download Ollama, CLI reference, OpenAI compatibility and Pricing, Ollama, read Nov 6, 2026
• gemma4, qwen3.5, qwen3.8, gpt-oss and llama4, Ollama library, read Nov 6, 2026
• LM Studio is free for use at work, LM Studio, Jul 8, 2025
• llama.cpp, GitHub, read Nov 6, 2026
• Gemma 4 model overview, Google AI for Developers, read Nov 6, 2026
• Gemma 4: Byte for byte, the most capable open models, Google, Apr 2, 2026
• gpt-oss-20b, Hugging Face, read Nov 6, 2026
• Mistral Large 4 1T A52B, Hugging Face, read Nov 6, 2026
• Introducing Muse Glimmer, Meta, Aug 10, 2026
• Muse Spark Open, Meta, Oct 15, 2026
• Model leaderboard, Artificial Analysis, read Nov 6, 2026
• Local AI Explained: How to Run AI Models on Your Computer, Tech With Tim, Aug 31, 2026 (video)