Self-Improving Agents Have an Overfitting Problem

The idea of a self-improving AI agent is intuitively compelling.

An agent attempts a task, observes what went wrong, modifies its own system, tries again, and gradually becomes better. Repeat the loop long enough, and we might expect increasingly capable agents to emerge.

But there is a subtle problem hidden inside this story:

What if the agent is not actually becoming better? What if it is simply becoming better at the benchmark used to improve it?

A recent paper, RRSI: Regularized Recursive Self-Improvement of Agent Harnesses, takes this question seriously. Its central argument is simple but important:

Recursive self-improvement is itself an optimization process, and optimization over a finite benchmark can overfit.

This sounds obvious once stated. But it has major implications for how we should build self-evolving agents.

From Model Improvement to Harness Improvement

When we talk about improving an AI system, we often think about changing model weights through training or fine-tuning.

Modern agents, however, are more than their underlying models.

An agent can be thought of as two parts:

$$\text{Agent}=\text{Model}+\text{Harness}$$

The model provides the underlying reasoning and generation capability.

The harness is everything around it:

• system prompts,
• tools,
• control flow,
• memory,
• skills,
• context management,
• retry logic,
• subagents,
• and various configuration choices.

Two agents using exactly the same model can behave very differently because their harnesses are different.

A coding agent, for example, may succeed not because its model is smarter, but because its harness tells it to inspect relevant files first, run tests before submission, recover from command failures, manage long contexts carefully, and verify the final result.

This naturally leads to a new idea: instead of asking humans to manually improve these systems, why not let agents improve their own harnesses?

The loop looks something like this:

$$H_t \rightarrow \text{run tasks} \rightarrow \text{analyze failures} \rightarrow \text{propose edits} \rightarrow H_{t+1}$$

This is a practical form of recursive self-improvement. The model weights remain frozen, but the system that shapes the model's behavior evolves over time.

And this is where the overfitting problem begins.

A Benchmark Can Quietly Become a Training Set

Imagine that we have a benchmark with a fixed collection of tasks.

At each iteration, the agent:

1. runs on those tasks,
2. inspects its failures,
3. changes its prompt, tools, memory, or control flow,
4. runs on the same benchmark again,
5. keeps changes that improve the score.

After enough iterations, what exactly does the benchmark represent?

It is no longer just an evaluation set.

It has effectively become a source of training signal.

Even if we never update a single neural-network parameter, we are still performing adaptive optimization against a finite dataset.

That means the same distinction familiar from machine learning appears again:

$$\text{Benchmark performance} \neq \text{General capability}$$

An agent may improve dramatically on the tasks used during evolution while gaining little on unseen tasks.

In other words, recursive self-improvement can produce the agentic equivalent of training-set overfitting.

Three Ways Self-Improvement Can Go Wrong

The paper highlights several failure modes that make this especially likely.

1. Benchmark-Specific Adaptation

Suppose the evolution benchmark repeatedly contains a certain type of task.

The agent may discover a rule that works extremely well for that pattern.

Its benchmark score goes up.

But the new behavior may not represent a general capability at all. It may simply be a benchmark-specific heuristic.

Over many rounds, the harness can accumulate more and more of these local tricks.

The system appears to be improving, while its true transfer ability barely changes.

2. Chasing Noise

Agent evaluation is noisy.

The same harness can produce different trajectories across repeated runs. Tool execution, sampling, environment behavior, and even evaluators can introduce variance.

Suppose a modification changes a score from 85 to 86.

Was the modification actually useful?

Or was that particular rollout simply lucky?

If every small improvement is accepted as evidence, random fluctuations can become permanent architectural changes.

After many iterations, the harness may contain a large collection of mechanisms that were never truly beneficial.

3. Complexity Accumulation

There is another easy way to improve benchmark performance: spend more resources.

Add another verification step.

Add another reflection loop.

Add a memory system.

Add a subagent.

Add more context.

Add another retry.

Each mechanism might produce a tiny improvement.

If the only rule is “keep anything that increases the score,” the agent gradually becomes more expensive and more complicated.

Eventually we may obtain an agent that consumes dramatically more tokens and compute without becoming proportionally more capable.

This is not necessarily genuine intelligence improvement. Sometimes it is simply resource accumulation.

Regularizing Recursive Self-Improvement

The core idea behind RRSI is to treat harness evolution more like machine learning.

If optimization can overfit, then it needs regularization.

Importantly, the paper does not prevent the agent from modifying particular components. The agent can still change prompts, tools, memory, skills, subagents, context management, or control flow.

Instead, it regularizes how the search proceeds.

The method separates this into two questions:

What changes should we try?

and

Which changes should we permanently keep?

These correspond to proposal-side and selection-side regularization.

Regularizing What the Agent Proposes

One idea is to limit how many things a candidate can change at once.

Early in evolution, the system may be allowed to make several edits simultaneously to explore quickly.

Later, the edit budget shrinks until a candidate may change only one major mechanism.

This has an important benefit: attribution.

If a candidate modifies the prompt, memory, verifier, and retry logic at the same time, and the score improves, we do not know which change mattered.

If it changes only one mechanism, the evidence becomes much cleaner.

The system also records the full history of previous edits:

• what was changed,
• why it was changed,
• whether performance improved,
• whether cost increased,
• and whether the change was ultimately accepted.

Failed ideas therefore become negative evidence rather than being forgotten and rediscovered several rounds later.

There is also an exploration mechanism.

If the search stalls after repeatedly modifying the same part of the harness, the proposer can be pushed toward components it has not explored yet.

This matters because language models themselves can develop search habits. If rewriting prompts is easy, the system might keep rewriting prompts even when the real opportunity lies in tool design or memory.

Regularizing What the Agent Keeps

The second half of the problem is arguably even more important.

A proposal may look promising, but that does not mean it deserves to become part of the permanent harness.

One rule is to screen for benchmark leakage before evaluation.

If a candidate includes task-specific names, hard-coded values, benchmark-specific conditions, or suspicious special cases, it can be rejected before receiving a fitness score.

Another rule accounts for noise.

The system first estimates how much the score naturally varies when the harness does not change. Small improvements inside this noise range are therefore treated cautiously.

This prevents a sequence of random lucky results from gradually corrupting the system.

The method also considers cost.

A modification that produces a small improvement while increasing token usage by 80 percent should not automatically be considered progress.

The basic principle is:

A more expensive harness must earn its additional cost.

Finally, components that repeatedly fail to demonstrate value can be targeted for deletion.

This creates pressure not only to add useful mechanisms, but also to remove unnecessary ones.

The result is a much healthier optimization process:

$$\text{Improve} + \text{Verify} + \text{Control Complexity} + \text{Prune}$$

rather than simply:

$$\text{Score goes up} \Rightarrow \text{Keep the change}$$

The Most Interesting Experimental Result

The paper's most revealing result is not that regularized evolution improves benchmarks.

It is that unregularized evolution can improve the evolution benchmark more while generalizing less.

In one experiment, unregularized harness evolution achieved a higher score on the dataset used during evolution than the full RRSI system.

At first glance, that looks better.

But when evaluated on out-of-distribution tasks, the regularized system performed substantially better.

The unregularized system also consumed far more tokens.

This is almost the textbook signature of overfitting:

$$\text{Training score}\uparrow$$

while:

$$\text{Generalization gain remains small}$$

The implication is uncomfortable but important.

If we evaluate a self-improving agent only on the environment that drives its self-improvement, we may systematically overestimate how much it has actually learned.

A Deeper View: The Harness Is Becoming a Learnable Object

There is a broader idea behind this work.

Historically, prompts, tools, memory systems, and agent control flows were treated as engineering artifacts.

Humans designed them.

Humans debugged them.

Humans decided which mechanisms to add or remove.

But once an automated system begins modifying these components itself, the harness becomes something closer to a learnable object.

We now have:

$$\text{Harness} \rightarrow \text{Search} \rightarrow \text{Evaluation} \rightarrow \text{Selection} \rightarrow \text{New Harness}$$

At that point, many ideas from machine learning become relevant again:

• train-test separation,
• regularization,
• model complexity,
• exploration,
• noise calibration,
• credit assignment,
• held-out evaluation.

The surprising lesson is that these ideas do not disappear when we stop training neural-network weights.

They reappear one level higher.

Recursive Self-Improvement Needs Generalization

The popular mental model of recursive self-improvement is:

$$\text{Agent} \rightarrow \text{find weakness} \rightarrow \text{modify itself} \rightarrow \text{become stronger}$$

But the more realistic version may be:

$$\text{Agent} \rightarrow \text{observe benchmark} \rightarrow \text{modify itself} \rightarrow \text{fit benchmark better}$$

Those are not the same process.

A genuinely self-improving agent therefore needs more than the ability to change itself.

It needs mechanisms that distinguish real improvement from local adaptation.

That may turn out to be one of the central challenges of agent evolution.

The long-term goal should not be an agent that can endlessly optimize its own benchmark score.

It should be an agent whose changes continue to work when the benchmark changes, the tools change, the tasks change, or even the underlying model changes.

In that sense, the problem of recursive self-improvement is not simply:

How do we make agents change themselves?

It is:

How do we make agents change themselves in ways that generalize?

And that is a much harder—and much more interesting—problem.