Learning path Intermediate
Local LLMs: run models on your own hardware
Run language models on your own hardware, no API and no quotas: from Ollama to serving big models on a Mac, plus vector memory for RAG.
- 8 resources
- 4 views
- ~80 min
Run language models on your own machine — no API keys, no quotas and your data never leaves home. The path goes from installing Ollama to fine-tuning on an Apple Silicon Mac, and ends by giving your model memory with a vector database for RAG.
How to Install Ollama to Run LLMs on Your Computer
Ollama makes it trivial to run models like Llama 2 or Mistral on your own computer: one binary, one command, and quantised weights downloading to disk with no compilation required. Covers installation on macOS, Linux, and Windows with an honest look at what local inference can and cannot do compared to frontier models.
How to Install Ollama on macOS with Apple Silicon
Installing Ollama on an Apple Silicon Mac is as simple as running one Homebrew command. Then pick a model based on available RAM (Phi-3 for 8 GB, Llama 3.1 8B for 16 GB) and expose the local, OpenAI-compatible HTTP API on port 11434 to plug it into your own applications.
Gemma 4 locally, which size fits your GPU
Gemma 4 ships in five sizes under Apache 2.0. Quantised to 4 bits the weights run from 2.9 GB on the E2B to 17.5 GB on the 31B according to Google's own table, so your memory picks the size. And the 256K window adds another 10 GiB of cache on top.
Local multimodal models: the VRAM arithmetic on a 16 GB card
Qwen3-VL 8B at Q4_K_M is 4.68 GiB of weights plus a 1.08 GiB vision tower, and its KV cache costs 144 KiB per token. At 8,192 tokens of context the total is around 7 GiB and fits comfortably in 16 GB; at the native 256K it would ask for 36 GiB of cache alone and fits on no consumer card.
What is oMLX and how it differs from MLX, Ollama and LM Studio
oMLX is a local inference server for Apple Silicon Macs that wraps Apple's MLX framework in a web process and exposes it through the OpenAI and Anthropic APIs. It adds continuous batching, on-disk KV caching and several models in memory at once, all driven from the menu bar.
How to install and tune oMLX on M5 Max 128 GB
Recipe for oMLX on a Mac M5 Max with 128 GB: install, TurboQuant at 3.5-bit, the Qwen 3.6 35B-A3B model stack, Claude Code wiring and our own benchmarks. Revised for oMLX 0.6.4, with what changed since 0.3.8.
How to Install PostgreSQL with pgvector Step by Step
This guide installs PostgreSQL 16 with pgvector on Debian or Ubuntu using the official PGDG repository, creates a dedicated role and database, tunes memory for production, and explains when the HNSW index beats IVFFlat depending on vector volume and the available maintenance window.
Local LLM Calculator
Will that model fit your GPU? Is self-hosting cheaper than the API? Estimate the VRAM and the cost break-even point.