Espresso is an open-source Swift library that runs transformers on the Neural Engine (ANE) of Apple Silicon Macs without using Core ML. It writes each layer’s program in MIL, Apple’s intermediate language, and compiles it with two private macOS classes that its author, Christopher Karani, documented through reverse engineering. This article covers what it does differently, how it works, what its numbers against Core ML measure, what happens with a real model like Qwen2.5 and the risks of relying on private APIs. We have not run it on a Mac of our own: everything comes from the repository and its history, reviewed on 25 October 2026. There is also a Spanish version of this article.

Key takeaways

  • Espresso compiles MIL programs for the ANE with the private classes _ANEClient and _ANEInMemoryModel, once per model, and keeps the KV cache in IOSurface buffers shared with the CPU.
  • On a 6-layer test model on an M3 Max, Espresso’s fused path takes 1.93 ms per token (519 tokens/s) and Core ML takes 6.58 ms (152 tokens/s): 3.41 times faster.
  • With Qwen2.5-0.5B, 10 of 12 sequences and 341 of 384 tokens match the PyTorch reference, but attention, RoPE and the output layer run on the CPU, and the project gives no speed figures.
  • Qwen2.5-0.5B’s output layer takes 272 MB and the SRAM available on the ANE is about 32 MB, so it does not fit.
  • An app that uses Apple’s private APIs will not get into the App Store, and a macOS update can break it.

What Espresso is and what it does differently

Espresso is an MIT-licensed Swift 6.2 package that talks to the Neural Engine with no Core ML in between. The Espresso README[1] sums it up like this: "No CoreML in the hot path. No per-token recompilation." The project started on 4 March 2026 and has 392 commits, 173 stars and one published release, v0.9.0, from 12 August.

The Neural Engine is the neural network accelerator Apple builds into its chips next to the CPU and GPU. Apple describes it as "the energy-efficient and high-throughput engine for ML inference on Apple silicon". Its guide to deploying transformers on the Neural Engine[2] gives its FP16 peak: 0.6 TFLOPS on the 2017 A11 and 15.8 TFLOPS on the 16-core ANE of 2021. The only public way to use it is Core ML, and Core ML decides at run time which operations go to the ANE and which to the CPU.

The usual tools for running LLMs on a Mac, such as oMLX or Ollama with MLX, use the GPU through Metal. Espresso targets the ANE, which draws less power, and does it without the Core ML wrapper. The library has no third-party dependencies: only Foundation, Accelerate, IOSurface, Metal and Core ML.

How Espresso works under the hood

Espresso replaces Core ML’s compilation with its own and reuses the compiled program for every token. According to the README, the sequence is:

  1. The MILGenerator module writes each kernel’s MIL program as text, with variants that fuse up to three layers into a single dispatch to the ANE
  2. ANEInterop loads the private _ANEClient and _ANEInMemoryModel classes with dlopen and compiles that text into an ANE binary, which the system caches
  3. Inputs, weights and outputs live in IOSurface buffers, memory shared by the CPU and the ANE, so reading a result does not require a copy
  4. The generation loop compiles once and repeats the evaluation at each step, with the KV cache in IOSurface as well

Other projects have taken the same road. The Orion paper, published on arXiv on 6 March 2026[3], compiles MIL and evaluates it with _ANEClient. It catalogs 20 ANE constraints (14 of them previously undocumented) and generates more than 170 tokens/s with GPT-2 124M on an M4 Max.

The mdaiter ane repository[4] documents that, without the right entitlements, _ANEClient is created but compileModel: returns NULL with no error. Espresso uses the in-memory model path and states that it works on macOS 15 with no special entitlements; on iOS it does need them.

How much faster Espresso is than Core ML

Espresso generates 3.41 times more tokens per second than Core ML in the only test it publishes, and that test uses a toy model. The benchmarks latest.json file[5] is dated 17 March 2026. It measures a 6-layer Llama-style model trained on TinyStories (dimension 768, 12 heads, 32,000-token vocabulary, sequence length 256) on an M3 Max with macOS 15.0:

Token generation on an M3 Max Neural Engine, 6-layer model tokens/s · higher is better
Espresso, fused recurrent decode
519
Espresso, unfused transformer
153
Core ML, .cpuAndNeuralEngine
152
Show the data table
Execution pathtokens/s
Espresso, fused recurrent decode519
Espresso, unfused transformer153
Core ML, .cpuAndNeuralEngine152

Figures from benchmarks/results/latest.json in the Espresso repository (17 March 2026), 6-layer stories110m model, macOS 15.0. Not measured by jacar.

The chart shows where the gain comes from. Without layer fusion, Espresso performs the same as Core ML (153 versus 152 tokens/s). The advantage appears with the path that fuses three layers per dispatch to the ANE and also computes the final classifier on the ANE.

The README itself scopes its numbers: they are not a comparison with 117-million-parameter GPT-2 or with llama.cpp on Metal, and the model is "a research / demo artifact family, not a pretrained production model". One detail that builds confidence is that the project’s continuous integration fails if the README table stops matching latest.json.

What happens with a real model like Qwen2.5

With Hugging Face models, Espresso splits the work between the ANE and the CPU, and its documentation avoids speed figures. The Qwen2.5-0.5B parity report[6] details what runs where:

Stage Where it runs Why
Q, K and V projections ANE One kernel per layer
SwiGLU feed-forward network ANE Fused kernels
RoPE, attention softmax and KV cache CPU Sequential, cache-bound work
Output layer (151,936 × 896) CPU 272 MB against about 32 MB of ANE SRAM

With that split, 10 of 12 test sequences and 341 of 384 tokens match PyTorch in FP32 with greedy decoding. The two sequences that diverge do so because of the ANE’s FP16 precision, with errors of up to one logit, and in both cases they pick the reference’s second-ranked token. The document closes with: "That is one model, greedy, fp16, measured — not a general-model claim, and not a speed claim."

The Qwen2.5-1.5B report[7] adds the hardware limits the author ran into:

  • Compile budget: each process can only compile a finite number of ANE programs, and 720 compilations would exhaust it
  • Layer widths: Qwen3-0.6B piled up 156 compile retries and 195 failures; Qwen2.5-0.5B, whose dimensions are all multiples of 64, compiles its 24 layers without a failure in about 20 s
  • FP16 overflow: in the 1.5B model, RMSNorm’s mean of squares overflows FP16 when the residual RMS exceeds 256, and Espresso scales the residual by 1/4 before squaring it
  • Output layer: in the 1.5B model it takes about 467 MB and also stays on the CPU

How to try Espresso on your Mac

The demo installs with three commands and needs an Apple Silicon Mac, macOS 15 or later and Swift 6.0 or later. These are the README’s steps as of 25 October 2026:

git clone https://github.com/christopherkarani/Espresso.git
cd Espresso
./espresso doctor
./espresso prepare
./espresso

doctor checks that the machine has an ANE, the scripts and a usable Python. prepare downloads the GPT-2 demo weights and tokenizer, which needs network access and a Python with torch and transformers (Espresso can create its own virtual environment). The last command builds the project if needed and opens the text demo in the terminal.

To chat with a real model, the README converts Qwen2.5-1.5B-Instruct to the project’s .esp format and opens a chat:

python3 scripts/convert_qwen25_05b_to_esp.py \
  --model Qwen/Qwen2.5-1.5B-Instruct
./espresso chat --model \
  ~/Library/Caches/Espresso/qwen25-15b/Qwen2.5-1.5B-Instruct.esp

The conversion script downloads the model from Hugging Face if you do not have it and packs the weights. The chat footer shows tokens per second, time to first token and joules per token for each reply. With --vs mlx --greedy it opens a second pane running MLX on the same model, which is the comparison worth making on your own machine.

Risks of using the Neural Engine’s private APIs

Espresso’s main risk is the same thing that gives it its edge: it depends on classes Apple neither documents nor guarantees. The README says so plainly: "Apps using private ANE APIs (_ANEClient, _ANEInMemoryModel) will be rejected." These are the practical consequences:

  • App Store: an app that links these classes will be rejected in Apple’s review
  • macOS updates: Apple can change or remove private classes in any release without notice
  • iOS and tvOS: they do not work as is, because private API entitlements differ per platform
  • Precision: the ANE computes in FP16, and in large models the errors add up layer by layer
  • Maturity: the latest published release is v0.9.0, yet the benchmark table cites an "Espresso 1.1.0" that does not exist as a release

There is one more detail worth correcting if you read the README. Its compatibility table credits the M3 with an "18-core ANE" and the M4 with a "38-core" one. Those are the trillion-operations-per-second (TOPS) figures Apple published for those chips, not cores, and the project’s own latest.json describes the M3 Max ANE as 16-core.

Espresso versus Core ML, MLX and llama.cpp

Espresso does not compete today with GPU tools for running large LLMs; it is a research project about the ANE. This table sums up the differences:

Criterion Espresso Core ML MLX and llama.cpp
Compute unit ANE, with attention on the CPU Whatever Core ML picks GPU through Metal
API Apple private Public Public
App Store safe No Yes Yes
Tested models 6-layer demo, Qwen2.5 0.5B and 1.5B Any model converted to Core ML Thousands on Hugging Face
Precision FP16 FP16 on the ANE Quantized, FP16 or BF16

If you want to run an 8B or larger model on your Mac, stay with MLX or with quantized models in llama.cpp. Espresso makes sense if you research the ANE, if you care about per-token energy for a small model, or if you want to see how to program an accelerator Apple keeps closed.

Frequently asked questions

Does Espresso work on an Intel Mac?

No. Intel Macs have no Neural Engine, and Espresso needs Apple Silicon (M1 or later) with macOS 15 or later.

Can I use Espresso in an App Store app?

No. The README itself warns that Apple rejects apps that use _ANEClient or _ANEInMemoryModel. For a published app, the route is Core ML.

Is it faster than MLX for chatting with an LLM?

The project does not claim so. Its only speed figure compares against Core ML on a 6-layer model, and for Qwen2.5 it only publishes parity. The chat includes a --vs mlx mode so you can measure it yourself.

Conclusion

Espresso shows that a Mac’s Neural Engine can run a transformer from Swift without Core ML, and that fusing layers makes it 3.41 times faster on a small model. With real models the ANE only keeps the projections and the feed-forward network, and its roughly 32 MB of SRAM leaves the output layer on the CPU. If you have an Apple Silicon Mac, run the demo and the chat with --vs mlx to measure it on your machine, but do not put it in an app headed for the App Store.

Sources: [1] Christopher Karani, Espresso repository[1], [2] Espresso, benchmarks/results/latest.json[5], [3] Espresso, Qwen2.5-0.5B parity report[6], [4] Espresso, Qwen2.5-1.5B parity report[7], [5] Apple Machine Learning Research, Deploying Transformers on the Apple Neural Engine[2], [6] Ramchand Kumaresan, Orion on arXiv[3], [7] mdaiter, ane repository[4].

Sources

  1. Espresso README
  2. guide to deploying transformers on the Neural Engine
  3. Orion paper, published on arXiv on 6 March 2026
  4. mdaiter ane repository
  5. benchmarks latest.json file
  6. Qwen2.5-0.5B parity report
  7. Qwen2.5-1.5B report