Agent Lightning: reinforcement learning for agents without rewriting them

You have an agent in production, it works reasonably well, and it keeps failing at the same kind of task. Someone told you reinforcement learning would fix it, and your sensible answer was that you are not going to rewrite the agent to find out. Agent Lightning, the framework Microsoft Research took to version 1.0 on 17 August 2026, exists for exactly that objection.

Key takeaways

• Agent Lightning applies reinforcement learning to the model behind your agent without touching its tools, its context or its control flow.
• The only change on the agent side is where its OpenAI client points: at the Agent Lightning proxy instead of the model.
• Version 1.0 is a full rewrite, MIT licensed, around 3,500 lines of Python, with a mode that runs every rollout as a Kubernetes Job.
• The published result: Qwen3.5-9B goes from 41.8% to 56.4% on SWE-bench Verified with 6,000 training examples.
• The plumbing is free. You write the reward function yourself, and that is where the real work sits.

What problem Agent Lightning solves

Training an agent with RL has always meant putting the agent inside the training loop. The RL engine wanted to own the interaction with the environment, so you rewrote the agent in the engine’s terms: its tools, its context management and its retries moved into training code. What you ended up training was a simplified version of the agent, not the one you had deployed.

Agent Lightning inverts that relationship. The agent still owns the loop, with the same harness you run in production, and the trainer only observes. The team puts it plainly in the v1.0 release notes: "Agents interact with the model through the Agent Lightning v1.0 proxy with ZERO changes, while keeping tools, context, control flow, and environments in the loop". That sentence is the whole pitch, and it is why the rest of the design revolves around a proxy.

The project appeared in Microsoft’s repository in June 2025 and now carries 17,917 stars and 1,573 forks. The first release shipped in August 2025; v1.0.0 was published on 17 August 2026 and v1.0.1, the current stable tag, a week later.

How the proxy interposition works

The system has three pieces. An API Gateway that stores rollouts, model endpoints and events, and also acts as an OpenAI-compatible proxy. A Rollout Controller that launches each agent execution as a local subprocess or as a Kubernetes Job. And a trainer built on verl and vLLM that turns what was recorded into policy updates.

The trick is in the URL. The agent does not talk to the model: it talks to a Gateway address with the rollout identifier encoded inside it.

POST /proxy/rollout/{rollout_id}/attempt/{attempt_id}/mode/train/openai/v1/chat/completions

Because the identifier travels in the path, every model call is bound to its execution with no bookkeeping on the agent side. The Gateway forwards the request to the registered endpoint and records the prompt token IDs, the response token IDs and the chosen-token log probabilities.

Reading server/proxy.py shows two details the documentation does not advertise: the proxy rewrites the temperature (1 for training, 0.7 for validation) and adds return_token_ids. The trainer then receives the exact tokens the model produced rather than an approximate retokenization. It also takes a SHA-256 hash of the rollout ID to pin each execution to one inference server and reuse the prefix cache.

On the agent side, all of this comes down to two lines. In the Calc-X example bundled with the repository, an AutoGen agent with MCP tools connects like this:

model_client = OpenAIChatCompletionClient(
    model="auto",
    base_url=os.environ["AGL_OPENAI_BASE_URL"],
    api_key=agl_key,
)

What harnessed agentic RL means

The v1.0 technical report calls this approach harnessed agentic RL and defines it without ambiguity. The deploy-time harness participates directly in the model’s post-training. So the harness, rather than the training engine, owns the environment interaction loop, while the trainer sees only sequences of request-response pairs.

The difference from retraining the model directly is one of subject. Classic fine-tuning optimises the model against a dataset. Here you optimise the model against the behaviour of the whole system: if your agent retries three times, halves its context and calls four tools, the model learns to play in that specific environment. Which also means a change to the harness invalidates part of what was learned.

The price of this architecture is a set of new problems the authors list honestly: retokenization, sample merging, advantage calculation, loss normalisation and backend scheduling. The trainer merges consecutive calls only when their token histories are exactly continuous, and computes advantages at the rollout level.

What version 1.0 changed

v1.0 is not an incremental release: the repository warns that earlier versions live on a separate v0.x branch. Three changes matter.

The first is size. The repository front page presents it as a 3,500-line framework. I measured it on a clone of main: the agentlightning package is 27 Python files, 4,771 physical lines and 3,835 excluding blank lines and comments. The round headline number holds up, and for an RL framework it is still tiny.

The second is Kubernetes. The controller can create one Job per rollout from a Jinja2 template you supply, with deterministic names of the form agl-rollout-<id> and the label app.kubernetes.io/managed-by=agentlightning. It isolates agent dependencies and runs rollouts concurrently on your own cluster instead of depending on a commercial sandbox service.

The third is the complete coding-agent example, with its data pipeline, its reward-hacking defences and its scripts. That is what the headline number rests on.

What you actually need to run it

This is where it pays to come down from the press release to the hardware. The quick start and the coding-agent example describe two different worlds:

Installation is not painless either. It needs CUDA 12.9 or 13.0 and a pinned verl and vLLM combination (verl 0.8.0 with vLLM 0.20.2, or verl 0.7.1 with vLLM 0.12.0). It also needs a source build of flash-attn 2.8.3 that the documentation itself puts at 10 to 30 minutes depending on available cores. The model also has to be open-weight and served by you: there is no way to run RL against a closed vendor endpoint.

Why the reward is the hard part

This is the part the press coverage skips. The framework gives you an events endpoint; the function that decides whether an execution went well is yours to write. In the Calc-X example, the agent finishes with a numeric comparison and a single POST:

httpx.post(
    event_url,
    json={"event_type": "reward", "data": {"value": reward}},
    headers={"Authorization": f"Bearer {agl_key}"},
).raise_for_status()

So the "zero changes" claim is exact for the harness and false for the reward. There is still a line to add, and behind that line, the most delicate decision in the project.

When the reward is weak, the agent learns to exploit it. The coding-agent documentation treats this as a first-order risk. Before the run starts, the harness moves the .git directory outside the visible testbed and blocks commands that invoke Git, install packages, download files or modify the test harness.

On the network side the recommendation is explicit: "we strongly recommend adding a Kubernetes network policy that denies all outbound traffic from agent pods except connections to the AGL Gateway. Without this restriction, an agent may retrieve upstream source code or other external information and obtain reward without solving the task as intended".

Preparing the data is no small matter either. To reach the 6,000 headline examples they started from SWE-smith’s 59,136 executable tasks across 128 Python repositories. They removed 18,033 with an empty problem statement and 1,265 with a missing branch, and dropped anything needing more than 200 tests.

They probed every remaining candidate four times with the base model and kept the mixed-difficulty ones, plus a thousand impossible tasks so the set was not biased. That work, and not the installation, is the real cost of the method.

If you are not yet measuring what your agent does, start there rather than with RL. An evaluation suite with DeepEval or promptfoo and traces in Langfuse are what tell you whether the reward you wrote measures what you think it measures.

Which agent frameworks it works with

The v1.0 answer is awkward to market and comfortable to use: any framework that accepts a custom base_url on its OpenAI-compatible client. The v1.0 README no longer publishes a list of supported frameworks, because the contract is the Chat Completions API rather than the framework.

The bundled examples use AutoGen with MCP tools and the plain openai client. The original 2025 paper, which described the earlier design, did name LangChain, the OpenAI Agents SDK, AutoGen and agents written from scratch. If your agent talks to the model through a client whose base URL you can change, it fits. If your agent buries the model call inside a service that will not let you touch that URL, it does not.

It is worth adding that the idea is no longer exclusive. The v1.0 abstract itself acknowledges that the disaggregated architecture they introduced has been adopted by verl Uni-Agent, AReaL 2.0, slime and Polar.

When RL is the wrong tool

It is almost never the first thing to try. Before standing up a cluster with four B200s:

1. Write the evaluation. If you cannot score a run automatically and reliably, you cannot train on it either.
2. Try better instructions and examples in the prompt. Free, reversible, and it fixes most formatting and judgement failures.
3. Try a larger model. Usually cheaper than a training campaign.
4. Try supervised fine-tuning on good trajectories. It needs demonstrations rather than a scoring function, and it is far more stable.
5. Check that the failure is the model’s. If your agent falls over on timeouts or mishandled retries, what you need is durable execution with Temporal, not reinforcement learning.

RL pays off when the failure is a policy problem, you have enough examples with an automatic and verifiable signal, the model is open-weight, and you have GPUs. Outside that intersection, everything else on the list returns more per euro spent.

Frequently asked questions

Can I train a closed vendor’s model with Agent Lightning?

No. The trainer updates a policy served by vLLM on top of verl, so the model has to be open-weight and running on your own hardware. With a closed endpoint you can use the proxy to observe, but not to train.

Is it true that nothing in the agent has to change?

Almost. You change the base URL and the key on its model client, and you add one reward report at the end of the run. Tools, context, retries and control flow stay as they were.

Do I need Kubernetes to try it?

No. The controller has a local mode that launches each rollout as a subprocess, and the quick-start example fits on a single machine with one A100. Kubernetes comes in when you want isolated dependencies and concurrent rollouts.

Conclusion

Agent Lightning solves one concrete, bounded problem well: removing the rewrite that until now made it impractical to try reinforcement learning on an agent that already works. The proxy is a clean idea, the 3,500 lines are verifiable, and the Kubernetes mode turns rollouts into something your cluster already knows how to schedule.

What it does not solve, and never claims to, is the question of what it means for a run to have gone well. That one stays yours, and it is why a team does better to deploy their agent properly and measure it before buying GPUs. The Spanish version of this article is at Agent Lightning: entrenar agentes con refuerzo.

Sources

1. Agent Lightning v1.0: Towards Harnessed Agentic RL, technical report
2. microsoft/agent-lightning, v1.0.0 release notes
3. Agent Lightning, components and rollouts documentation
4. Agent Lightning, coding agent example
5. Agent Lightning: Train ANY AI Agents with Reinforcement Learning
6. Microsoft Research, reinforcement learning without code rewrites