Category
AI without the hype: models, agents and use cases that work in production.
Related paths
On an arm64 CPU with no GPU, llama.cpp 0.5.0 served Qwen3-1.7B as a Q8_0 GGUF at 72.7 tokens/s per request and 236.9 tokens/s with 8 concurrent requests. vLLM 0.30, which only loads BF16 on Arm, reached 10.1 and 68.3 tokens/s and used 11.4 GiB of RAM against 7.0 GiB.
oMLX serves text-to-speech (TTS) models on an Apple Silicon Mac through the same API as OpenAI. With POST /v1/audio/speech you request audio in a preset voice, a voice described in plain language or a voice cloned from a few seconds of reference. On an M5 Max, Qwen3-TTS 1.7B generates each second of audio in 0.33 s.
oMLX 0.7.2, released on 21 October 2026, one day after 0.7.1, speeds up DeepSeek V4.1 on the oQ2e to oQ4e quants, turns on Lightning MTP in multi-Mac clusters, runs Qwen3.8-Flash-Next GGUFs on Splash and stops re-prefilling the whole conversation on every Claude Code turn. You upgrade it from the app, Homebrew or source.
Since Ollama 0.40, released on 25 September 2026, models with a supported architecture run on MLX instead of llama.cpp on Apple Silicon Macs. Ollama picks the model variant at download time, and ollama ps shows in its RUNNER column which engine serves it. On Linux and Windows llama.cpp is still the default.
llama-server prints error: invalid argument: --mlock because build b10875, from 9 September 2026, deleted --mmap, --no-mmap, --mlock and --direct-io. Replace them with --load-mode: --mlock becomes mmap+mlock, --no-mmap becomes none and --direct-io becomes dio. The LLAMA_ARG_MLOCK and LLAMA_ARG_MMAP variables now do nothing, with no warning.
Clef and Clef-flash are the open decision models Cloudflare released on 1 October 2026: instead of text they return a choice with its probability. On Ollama 0.35.1 you call them through /v1/systemone. Clef-flash labelled 23 of 24 support tickets correctly in my test, and the only miss had the lowest confidence.
September 2026 produced few open models of home-machine size. The one worth disk space is Xiaomi's MiMo-V2.6-Distill-Qwen-9B, MIT licensed: in my CPU tests it got tool calls and JSON right. MiniCPM5-2B remains the small option, and DFM Mimir is only useful in Danish or English.
Soup is an open-source CLI (Apache 2.0) that covers LLM post-training end to end: it decides whether training is worth it, writes the config, trains with LoRA, DPO or GRPO and returns a SHIP or DON'T SHIP verdict. I tested version 0.76.1 on 15 October 2026 on Linux arm64 with no GPU.
Since oMLX 0.7.1.dev1, one server on your Mac serves Qwen on /v1/chat/completions and the Clef decision models on /v1/systemone. On an M5 Max, Clef-flash decided each ticket in 111 ms and, with Qwen writing the reply, triage got 39 of 40 labels right against 34 with Qwen alone.
llama-server, the llama.cpp server, exposes /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models in OpenAI's format, so the official SDK works by changing base_url. I tested v0.5.0 in Docker: tools, schema-constrained JSON and Open WebUI work, but the context is split across slots and a named tool_choice is ignored.
To install gsd-pi, the Open GSD terminal agent, run npx @opengsd/gsd-pi@latest with Node.js 22.18 or later. It works without an API key: with a models.json pointing at llama.cpp and a 64k context, a 4B Qwen3.5 finished a task with 12 passing tests and a commit. gsd-core, by contrast, installs inside Claude Code or OpenCode.
garak 0.17 is NVIDIA's LLM vulnerability scanner: it fires prompt injection, jailbreak, encoding and data-leak probes at your model and counts the failures. Against gemma3 1B on Ollama, on CPU, prompt injection succeeded 66.67% to 93.33% of the time, and a defensive system prompt did not stop it.
Ollama 0.34.1 through 0.34.4 reject with HTTP 400 any request to /api/chat or /api/generate that carries typical_p in options, even at 1.0. The fix is to drop the field in the client, go back to 0.34.0, use the /v1 endpoint, or move to 0.40.0, which only logs a warning.
Without a GPU, on an 18-core arm64 CPU with llama.cpp v0.4.1, Maple-Preview generated 172.84 tokens/s on 6 threads: twice MiniCPM5-2B and 3.6 times Spark-X2.5-4B. The trade-off is 5.79 GiB of RAM and reasoning it cannot switch off. MiniCPM5 fits in 3.27 GiB and Spark writes the best Spanish. All three made facts up.
Since Ollama 0.34.1, ollama create no longer converts or quantizes safetensors weights. To import a Hugging Face model, download it with hf download, convert it to GGUF with llama.cpp's convert_hf_to_gguf.py, quantize it with llama-quantize and build it from a Modelfile. With MiniCPM5-2B on a CPU, the three stages took 35 s, 21 s and 2 s.
Colibri is an Apache 2.0 inference engine written in C that runs Mixture-of-Experts models of up to 2.8 trillion parameters without loading them whole: it keeps the dense part in RAM and reads from the SSD only the experts the router picks for each token. That is why its speed depends on the disk, the cache and free RAM.
ThinkingBox is an MIT-licensed Microsoft sandbox that runs an agent against stateful MCP tools and judges it by the side effects it leaves in the system, not by what it claims to have done. Its companion benchmark covers 507 business tasks, each repeated twenty times.
Agentic search means the model plans its own queries against the repository using grep, file listings and a language server, instead of querying an embedding index built beforehand. Claude Code and Cline work this way by design, yet Cursor and Devin Desktop still index, and their own documentation says so.
Agent Lightning is the Microsoft Research framework that applies reinforcement learning to the model behind an agent you already wrote. The agent points its OpenAI client at the Agent Lightning proxy instead of the model, and the proxy records requests, responses and rewards while the agent keeps its tools and control flow.
The 2026-07-28 revision turns MCP into a stateless protocol. The initialize exchange and the Mcp-Session-Id header are gone, and the protocol version plus client capabilities now travel inside the _meta field of every request. In exchange, Roots, Sampling and Logging are deprecated with a twelve-month window.
Qwen3-VL 8B at Q4_K_M is 4.68 GiB of weights plus a 1.08 GiB vision tower, and its KV cache costs 144 KiB per token. At 8,192 tokens of context the total is around 7 GiB and fits comfortably in 16 GB; at the native 256K it would ask for 36 GiB of cache alone and fits on no consumer card.
The Apple Watch Ultra 3 does not run Siri AI on its own. The model runs on the paired Apple Intelligence iPhone and on Private Cloud Compute, and Apple is not shipping Siri AI in the European Union because of its Digital Markets Act dispute. In the EU the watch keeps Workout Buddy, on-device Siri and its health alerts.
Gemma 4 ships in five sizes under Apache 2.0. Quantised to 4 bits the weights run from 2.9 GB on the E2B to 17.5 GB on the 31B according to Google's own table, so your memory picks the size. And the 256K window adds another 10 GiB of cache on top.
August 2026 produced about a dozen models with genuinely downloadable weights, and several names that exist only as a hosted API. The ones that change what fits on your own machine are Muse Glimmer 30B and Qwen3.8-27B, both Apache 2.0, plus Ling-3.0-tiny for smaller hardware.
Muse Code is the terminal coding agent Meta Superintelligence Labs released in beta on 5 August 2026. It runs on Muse Spark 1.2, coordinates persistent background subagents, appends every model call and every edit to a local event log, and installs with a single command on macOS and Linux.
Between 1 June and 28 August 2026, Claude Code shipped 81 releases, from 2.1.160 to 2.1.251, carrying 149 additions. Four of them change how you actually work: Claude Opus 5 as the default, cross-session messaging, self-hosted environments, and credential masking in the sandbox.
oMLX is a local inference server for Apple Silicon Macs that wraps Apple's MLX framework in a web process and exposes it through the OpenAI and Anthropic APIs. It adds continuous batching, on-disk KV caching and several models in memory at once, all driven from the menu bar.
FreeToken is an open-source inference engine, released by UC Berkeley and UT Austin researchers in August 2026, that splits a MoE model's experts across GPU, CPU and system memory according to each machine's measured bandwidth. It serves models from 35B to 753B parameters on consumer hardware.
OpenRouter is a hosted gateway that gathers more than 500 AI models from over 80 providers behind a single OpenAI-compatible API. You change the base URL and use one key and one balance. You pay each model's rate with no markup, plus 5.5% when you buy credits, and gain automatic routing and failover.
browser-use is an open-source Python library that lets an AI agent drive a web browser the way you do: it opens pages, reads the DOM, clicks and fills in forms. It was born on Playwright and in 2025 switched to speaking Chrome's CDP protocol directly for speed. This guide explains how it works and when to use it.
Firecrawl is an open-source API that turns any web page into clean Markdown, ready for a language model. With scrape and crawl endpoints plus schema-guided JSON extraction, it gives your AI agent reliable web data. This guide covers its formats, how to self-host it with Docker and when it beats a scraper of your own.
Composio connects your AI agent to more than a thousand applications (GitHub, Slack, Gmail, Notion) through pre-authenticated tools and managed OAuth authentication. Instead of writing each integration by hand, you request the tools by name and the agent acts on behalf of each user. This guide explains what it is, how it handles authentication and how to plug it into your agent framework.
Dify is an open-source platform for building AI applications and agents, with a visual workflow canvas, prompt management, a RAG knowledge base and LLMOps layers. You can self-host the whole thing with Docker Compose on top of Postgres, Redis and a vector database. This guide explains how to deploy it and when it beats Flowise and Langflow.
Flowise is an open-source tool that lets you build AI agents by dragging nodes onto a visual canvas, with barely any code. It is built on LangChain.js and you can self-host the whole thing with a single Docker container. This guide covers how to deploy it, the difference between chatflows and AgentFlow v2, and how it compares to Langflow and Dify.
Langflow is an open-source tool for building AI agents and workflows by dragging blocks onto a visual canvas, with barely any code. You can self-host it with Docker, wire up models, tools and vector databases, and publish every flow as an API or as an MCP server. This guide explains how to deploy it and build your first agent.
Your own MCP server exposes your data and functions to any AI assistant through the Model Context Protocol. With the official Python SDK and its FastMCP class you write a server in about twenty lines: you define tools, resources and prompts as type-annotated functions, and you connect it to Claude, Cline or Goose over stdio or HTTP.
Durable execution lets an AI agent survive crashes, restarts and API rate limits without losing its progress. Temporal applies this model: your logic lives in a workflow that resumes exactly where it stopped, and every model or tool call runs as an activity that Temporal retries for you automatically on failure.
Deploying an AI agent to production means turning your local script into a containerised service, with external state, observability and managed secrets. In this guide you package an agent inside a minimal Docker image, expose an HTTP endpoint with FastAPI and add health checks, retries and rate limits so it survives real traffic.
E2B is open-source infrastructure that runs your agent's generated code inside isolated Firecracker microVMs, each with its own Linux kernel. It boots in about 150 ms, exposes a stateful interpreter from its Python SDK and can be self-hosted with Terraform. This guide explains why an agent needs a sandbox and how to use E2B.
Prompt caching stores the stable prefix of your prompt (instructions, documents, tools) so it isn't reprocessed on every call. Anthropic and Gemini discount those tokens by up to 90%; OpenAI discounts by 50% on classic models and up to 90% on GPT-5.6. This guide covers each provider, how to structure the prompt, and how to cache on your own machine with vLLM.
promptfoo is an open-source tool to test and evaluate prompts, agents and RAG systems with declarative YAML configs. It compares models like GPT, Claude or Gemini in one matrix, scores answers with automatic assertions and adds red teaming with over fifty vulnerability types. It plugs into your CI, and OpenAI acquired it in March 2026.
DeepEval is the open-source framework for evaluating AI systems as if you were writing Pytest tests: you define cases, pick research-backed metrics (G-Eval, faithfulness, tool correctness) and get a score from 0 to 1 that passes or fails each response, both on your machine and in continuous integration.
Helicone is an open-source LLM observability platform you integrate by changing a single line: your client's base URL. It logs cost, latency and tokens for every call, adds caching and rate limiting, and you can self-host it with Docker. It is Apache-2.0 and has nearly 6,000 GitHub stars, though since March 2026 it sits in maintenance mode.
Langfuse is an open-source platform to observe, debug and evaluate AI applications and agents. You can self-host it with Docker Compose on Postgres, ClickHouse, Redis and S3 storage, and its Python SDK, built on OpenTelemetry, captures traces, spans and generations with their cost and latency. This guide explains how to deploy it and instrument an agent.
DSPy is a Python framework from Stanford that treats calls to an LLM as code: you define signatures with typed inputs and outputs, pick a module such as chain of thought, and let an optimizer write the prompts for you from examples and a metric. That way you compile programs instead of hand-tuning prompts.
Outlines is a Python library for constrained generation: it forces the model to produce output that conforms to a JSON schema, a regular expression or a grammar, with a guarantee by construction rather than statistics. It works with vLLM, Transformers, Ollama and llama.cpp, and its Rust engine adds barely any overhead per token.
Instructor is the most widely used Python library for getting reliable structured outputs from a language model: you define the result you expect as a Pydantic model, Instructor patches the client so the model honours it, and it retries on its own, with the validation error included, until it returns a valid, already-typed object.
Not every open model calls tools equally well: the Qwen3 family, Nous Hermes, Llama 3.1 and Mistral stand out because they were trained for it. To choose, check the Berkeley Function-Calling Leaderboard, make sure a parser exists for the model's template, and match it to your GPU's VRAM.
