Tag
Qwen3-VL 8B at Q4_K_M is 4.68 GiB of weights plus a 1.08 GiB vision tower, and its KV cache costs 144 KiB per token. At 8,192 tokens of context the total is around 7 GiB and fits comfortably in 16 GB; at the native 256K it would ask for 36 GiB of cache alone and fits on no consumer card.
Gemma 4 ships in five sizes under Apache 2.0. Quantised to 4 bits the weights run from 2.9 GB on the E2B to 17.5 GB on the 31B according to Google's own table, so your memory picks the size. And the 256K window adds another 10 GiB of cache on top.
August 2026 produced about a dozen models with genuinely downloadable weights, and several names that exist only as a hosted API. The ones that change what fits on your own machine are Muse Glimmer 30B and Qwen3.8-27B, both Apache 2.0, plus Ling-3.0-tiny for smaller hardware.
Function calling lets a model you run with Ollama on your own machine ask your code to call a function (check the weather, query a database) and use the result to answer. Ollama has supported tools since July 2024; in 2026 models such as qwen3 and llama3.3 do it with reasonable reliability.
Ollama 0.5 or newer runs Llama 3.3 70B and Mistral Large 2 locally on Ubuntu 24.04: Q4_K_M quantization lets a single NVIDIA GPU with 24 GB of VRAM, an RTX 4090 for example, handle the full model. This guide installs the drivers, sets up Open WebUI, and exposes the service behind Traefik with TLS.
Ollama became the standard for running large language models locally in 2024. It wraps llama.cpp in a single binary with Docker-style CLI and an OpenAI-compatible API. Phi-3 Mini runs in 4 GB; Llama 3.1 8B Q4 needs 6 GB. For production traffic at scale, vLLM remains the correct choice.
Installing Ollama on an Apple Silicon Mac is as simple as running one Homebrew command. Then pick a model based on available RAM (Phi-3 for 8 GB, Llama 3.1 8B for 16 GB) and expose the local, OpenAI-compatible HTTP API on port 11434 to plug it into your own applications.
LM Studio is a desktop app for Mac, Windows, and Linux that downloads and runs large language models on your own machine, with a polished chat interface and no terminal required. It includes an OpenAI-compatible API and RAG with your documents. For individual use it beats Ollama on user experience; for teams or production, OpenWebUI, vLLM, or TGI are the better fit.
Ollama makes it trivial to run models like Llama 2 or Mistral on your own computer: one binary, one command, and quantised weights downloading to disk with no compilation required. Covers installation on macOS, Linux, and Windows with an honest look at what local inference can and cannot do compared to frontier models.
