Tag
A caching proxy in front of a language model can cut the token bill significantly, but it introduces subtle risks if the design is not careful. Which cache types work in production, where the usual traps sit, and how to add them without degrading the experience.
Garnet is the open-source cache server Microsoft Research published in March 2024. Written from scratch in .NET 8, it speaks the Redis wire protocol, so existing clients connect unchanged, and it stores data through a hybrid memory-and-disk backend called Tsavorite. Core-affinity threading is what lets the Garnet cache outrun Redis on many-core hardware.
Dragonfly is a Redis-protocol-compatible cache built on a multithreaded shared-nothing design with one thread per data shard, so an eight-core node uses all eight cores where Redis uses one. Its fork-free snapshot algorithm keeps latency flat while persisting. Ordinary clients work fine; modules like RedisSearch and RedisJSON are where compatibility frays.
Valkey 8.1, released on March 31, is protocol and command compatible with Redis 7.x, so most existing clients connect without code changes. Swapping Valkey in for Redis is a realistic move once you plan the cutover: dump and restore is the simplest path, while zero downtime needs asymmetric replication. We moved our first production cluster two weeks ago.
Redis moved to dual SSPL/RSAL licensing in March 2024, no longer meeting the OSI open-source definition. Valkey emerged as a BSD 3-Clause fork backed by AWS, Google Cloud, Oracle, and the Linux Foundation, fully protocol-compatible with Redis 7.2. Migrating is almost always trivial: swap the binary or the Docker image.
Redis alone isn't a caching strategy, just an ingredient: picking the right pattern among cache-aside, read-through, write-through, and write-behind, sizing TTL to how fast data actually changes, invalidating explicitly for critical data, and mitigating thundering herd with jitter and locking are the decisions that actually matter in production.
