Who Should Manage an Agent's Context?

We gave an agent eight logs to audit. Halfway through, a line in one log changed a rule: from now on, report incident ids in lowercase. By then, the agent's context had already gone through OpenAI's server-side compaction once.

The final report came back in uppercase. Right after, we asked the same model to list the instructions in force. Its list ended with:

From log_5 onward, report incident ids in lowercase, ids only; this is the latest instruction changing the uppercase formatting rule.

How can a model recite a rule right after breaking it? We ran into this while comparing six ways to manage an agent's context in Kedi, our typed orchestration language for LLM workflows.

The short version

The agent compacting its own context did best. A fork of its own conversation writes the summary and chooses what stays verbatim. In two pre-registered tests on gpt-6-luna, it passed more runs than OpenAI's server-side compaction, which we call native: 26 of 30 against 19 of 30 on tasks whose instruction changes midway, then 15 of 15 against 8 of 15 on a longer task with new seeds.

Compaction did not lose the changed rule. In the first test and its supplement, all 15 failed native runs could still name it afterwards, and 9 of their reports had just broken it, as in the run above. The fork's failures looked the same; there were fewer of them.

The effect is shown on one model only, and on short tasks a simple reset was just as good. Every number below can be checked against the public raw records.

Compaction is a correctness problem

Long-running agents outgrow their context window, so the history gets compacted: older turns are dropped or replaced by a summary. Most discussion of compaction is about cost. This study is about what the agent does next.

Instructions do not all live in the system prompt. In real work they arrive mid-task: a reviewer changes a rule, a log puts a service out of scope, an incident is retracted. After compaction, such a change has to survive in whatever the compactor kept, and the agent then has to act on it. We expected the first step to fail, and pre-registered that expectation. In our runs the first step held; the second did not always.

So we asked two questions. Which way of compacting keeps a changed instruction working afterwards? And what does each way cost?

Six ways to manage a context

The strategies differ in one thing: who decides what the next request sees, and from what.

The fork is the conversation's own next request with one message appended: same model, same instructions, same tools, set to answer in text. The agent answers with a proposal: which exchanges to keep, and a summary of the rest. A1 writes that summary as structured JSON, A1-free as free text, and A1-fixed keeps the last two exchanges instead of choosing.

The runtime, not the model, has the last word on the layout. It rejects unknown ids, always keeps protected and unfinished exchanges, and restores the original order. A malformed proposal falls back to keeping the last four exchanges, so compaction never fails a run.

Every strategy is an ordinary Kedi history processor with the same trigger: an estimated 18K tokens. Building them surfaced two Kedi limits, which we fixed. A processor could not trim a tool loop that was still running, and native compaction was skipped when a run budget wrapped the model.

Both keep the prompt and the earlier summaries. They differ in what the compactor sees, who writes the summary and who chooses what stays verbatim.

Here is a real fork reply. It comes from an illustration run we made for this post, not from any test: A1-free on seed 106 of the owner-rule task, where a line in log 5 switches the owner to whoever owns the lowest-severity incident. At the fourth compaction, after log 7, the fork chose to keep only g0, the protected prompt (earlier summaries and the exchange in progress stay regardless), and wrote:

Template rules: exclude service legacy; report incident IDs uppercased, IDs only; owner is for the reported incident with the highest current severity unless superseded by a later CHANGE. Must read logs 1–8 in order, one call at a time, then fill report with incidents list and owner. Logs 1–4: inc-4349 ingest severity 7 owner ayse; inc-2408 payments severity 5 owner mehmet; inc-5527 storage severity 3 owner zeynep; inc-6388 payments severity 4 owner can. Log 5: inc-9943 storage severity 9 owner elif; OWNER RULE CHANGE says owner is for the reported incident with the LOWEST current severity. Log 6: inc-2337 service legacy severity 6 owner deniz; exclude it. Log 7: inc-1711 search severity 1 owner baran. No later change seen through log 7. Logs 1–7 were read; next read log_8, then report all eligible IDs in uppercase. Under current lowest-severity rule, owner among eligible reported incidents is baran (severity 1), assuming no later severity-1 or lower reported incident in log 8.

The summary carries the change and works out its consequence: so far the owner is baran. Log 8's incident had severity 2, so that stayed right. The report still named ece, the owner of that last incident. In the pre-registered test the fork passed this seed, and native failed it with the same wrong owner. A correct summary followed by a wrong report is the subject of Remembering is not applying.

How we tested

Every run is the same job: an agent audits incident logs and reports the incident ids plus one owner. It reads one log per read_log call, about 6.7K tokens each. The prompt fixes the rules: ignore the legacy service, report ids in uppercase, and name the owner of the most severe reported incident.

Then the logs change things. An UPDATE raises an old incident's severity, a RETRACT withdraws one, and a POLICY CHANGE adds a service to ignore, including incidents already read. The longest task has 20 logs, about 135K tokens, so every strategy compacts many times. In the held-out tasks, one CHANGE line in log 5 arrives after the first compaction. It flips the owner rule to the lowest severity, switches ids to lowercase, or narrows the report to three services.

Grading is deterministic and strict: exactly the right ids, ids only, in the required case, with the right owner. A seed generates the same scenario for every strategy, so runs are compared in pairs with an exact McNemar test. Differences come with 95% confidence intervals for paired proportions (Newcombe's method). The main runs used gpt-6-luna at reasoning effort medium.

We worked in phases:

• Phases 1 to 5 were exploratory. They shaped the design and are not evidence on their own.
• Before phase 6 we committed the hypothesis, the held-out seeds, the primary comparison and the analysis. Phases 9 and 10 were pre-registered the same way, and phase 9's three hypotheses get a Holm correction.
• A run lost to rate limits or outages got one rerun with the same seed. If that was lost too, it counts as missing, not as a failure.

Results

The fork passed more runs than native compaction in both pre-registered tests.

In the held-out test, 9 pairs of runs went the fork's way and 2 went native's: an advantage of 23 percentage points, with a 95% confidence interval of 3 to 42 points. The pre-registered one-sided test gives p = 0.033 (two-sided, 0.065; the notes on method at the end explain the choice).

The held-out test's third task, the scope change, was lost to provider usage limits. A supplement run outside the pre-registration pointed the same way: 13 of 15 against 11 of 15.

The second test used the 20-log task and new seeds. There the split was 7 pairs to 0, an advantage of 47 points (95% confidence interval 17 to 70), and it survives a Holm correction for phase 9's three hypotheses. Six of native's seven failures were policy violations: reports that still listed incidents from the service the policy change had excluded.

The fork costs more per run. Native used the least input of any compactor: 113K median tokens per held-out run against the fork's 159K to 168K, and 269K against 442K on the 20-log task. The fork also adds one request per compaction and writes about 1.7 times the output.

Per passing run, the gap closes. One correct report cost 191K input tokens with the fork against 204K with native on the owner-rule task, 195K against 194K on the format task, and 529K against 544K on the 20-log task. Median run times stayed within a few seconds of each other, from 31 against 35 seconds to 94 against 96. We report tokens and time, not dollars: most runs used a subscription quota, where list prices say little about the bill.

Remembering is not applying

After each held-out run we asked the model to list the instructions in force. Every failed native run named the change correctly: 11 of 11 in the test, 4 of 4 in the supplement. We had pre-registered the opposite, that compaction would lose the change. It did not.

Most of those reports still broke it: 9 of the 15, like the run at the top of this post (seed 103 of the format task). The other 6 applied the change and got an owner or an incident wrong. The fork's failures split the same way: all 6 named the change, and 4 had broken it.

One pattern we noticed only afterwards: on the owner-rule task, every report that broke the change named the owner of the last incident read. That held for all five of native's (in three of them it was also the old rule's answer), for both of the fork's, and for the illustration run above, right after a summary that named the right owner. Why a report breaks a rule the model still knows, the probe cannot tell.

No compaction fails differently. Its failed runs could not name the change at all (0 of 27), which looks like attention loss in a long context, not a compaction effect.

The gap between knowing a rule and applying it suggests a cheap repair, and it mostly works. In native+reassert the runtime restates each instruction change verbatim just before the current exchange. Policy violations fell from 11 to 4 on the 20-log task and from 8 to 0 on the 8-log task, while median input stayed at plain native's level. On the 20-log task it still trailed the fork: 9 of 14 runs passed, against 12 of 15.

Short tasks need none of this. On the 8-log task, a deterministic reset to a runtime checkpoint passed 13 of 14 runs at the lowest median input, 84K tokens; the fork passed 12 of 14. Over 20 logs the same reset fell to 4 of 15, with wrong owners after many resets, early stops and runs that hit the request limit. Carrying the facts is not the same as carrying a usable working state.

Why the fork holds up is not isolated. It differs from native in four ways at once: it sees the full context, chooses which exchanges stay, writes an explicit next step, and is written by the agent itself. An early ablation on the easiest task could not separate them, since every fork variant passed all its runs. Our hypothesis is that re-expressing state, rather than listing evidence, is what helps; the ablations that would test it are the next experiment.

Where it did not hold

The effect is shown on one model. We tried two others, and neither test could separate the strategies.

• gpt-6.1-terra at effort low sat at a floor. It passed 1 of 27 runs with the fork and 1 of 27 with native compaction. It treats instructions found in tool output as untrusted: asked afterwards, it said in 42 of 54 runs that the change was not in force. On the same tasks gpt-6-luna applied the change in 49 of 60 runs. This explanation came after the test and was not pre-registered.
• deepseek-v4.1-flash sat at a ceiling. On the 20-log task the fork passed 15 of 15, a stateless summary 14 of 15, and no compaction 15 of 15. With its 1M-token context it did not need compaction on this task. Compacting at 18K only doubled its median input, from 1.26M to 2.5M tokens.
• No final report obeyed a planted fake instruction. The logs held a marked, legitimate change and an unmarked fake one. The fork passed 13 of 15 against native's 11 of 15, which is not significant (p = 0.31). No final report followed the fake, for any strategy (0 of 15 each). We checked final reports only, not whether the fake entered an intermediate summary.

A deciding test on another model needs a harder task or an authorized channel for instructions. That was beyond this experiment's budget.

We also withdrew a claim. Early notes said the fork reads the provider's prompt cache in 5 of 6 runs; a rerun with raw records showed 2 of 6. We make no cache claim, and no result here depends on caching.

The other limits are the usual ones for a first study. The tasks are synthetic, with unambiguous evidence lines, and instructions arrive through tool output. The 18K trigger is artificial, and each comparison has 14 to 30 runs. The fork is also never the cheapest: it used 30 to 60 percent more input per run than native on every task.

What to take away, and how to check it

No single strategy wins everywhere; the session decides. From these tasks:

Two more lessons hold beyond the table. Tie the compaction trigger to the model's window: on a 1M-token model, an 18K trigger doubled the cost and bought nothing. And watch where instructions come from: one model applies a rule found in tool output, another declines it before compaction is even involved. A hybrid, native with restated rules and the fork only on long horizons, is a proposal we have not tested.

Everything is in kedi-lang/benchmarks: the harness, the raw record of every run, the pre-registrations, and one results file generated from the records. Any table rebuilds from the records with plain Python 3.12 and no dependencies: python -m agent_compaction.summarize. Rerunning the experiment needs Kedi itself, installed from git at a pinned commit. Kedi's repository stays private until its release; its documentation is public.

Notes on method

Grading. Besides a strict pass, each held-out report is checked for whether it applied the change: the owner of the lowest-severity incident (in any letter case), every id in lowercase, or only the three named services.

Tests and intervals. The held-out test was one-sided because its hypothesis named a direction. Intervals are 95% confidence intervals for a difference of paired proportions (Newcombe's method 10). For the held-out test the interval, 3 to 42 points, leaves out zero while the two-sided test (p = 0.065) does not reach 5%. The interval is a score approximation, not the inverse of the exact test, and near 5% the two can disagree; the pre-registered one-sided test is the one that decides.

The probe. After grading, one more turn in the same session asks for every instruction in force. Its answer is matched against keywords per task (lowest, lowercase, the three services), and it cannot change a run's result.

The illustration run. The fork reply in Six ways to manage a context comes from one run made for this post with the published harness, gpt-6-luna and Kedi at commit 3b90c18, not from any test. Its record and all four of the fork's replies are in records/examples, with a README.

Run-to-run variation. Pairing by seed holds the scenario fixed, not the model's behavior: each strategy ran once per seed, and a rerun of the same seed can end differently. The illustration run did. The fork passed seed 106 in the test and failed it in the rerun, which also used a later Kedi commit.

Pre-registrations. They were committed to the experiment's git history before their runs. That history was kept locally and published afterwards, so the commit dates rest on our word.