Blog
Notes on my latest research: self-improving coding agents, the knowledge graphs they build, and what an agent's stated values do to the code it writes. Three report a record: one agent's solver beat 21 published circle-packing records, an earlier one took the record at 27 circles and is credited for it on Packomania, and another beat the top human entry in an AtCoder contest, confirmed on the official judge. The rest is me checking my own results, including the ones that did not hold up.
-
My AI agent's new solver beat 21 circle-packing records at once Record
A self-improvement run built a solver that beats 21 live Packomania records. The follow-up run, warm-started from it, found nothing new.
-
My AI agent's solver beat the top human entry in an AtCoder contest Record
Eight self-improving agents on a real AtCoder task. The official judge scored the best one 573,184 against the leaderboard's 566,997.
-
My AI agent's values affected its code quality and performance on 2048
A single-run experiment said values drove generalization. Thirty runs found a large effect on capability, Cliff's delta +0.80, and a null on the unseen board.
-
Deriving useful values for AI agents from popular research papers
I derived an agent's values file from cited papers, then audited every citation: 22 of 22 real, none invented, no new bullets earned.
-
My AI agent given free rein over its values used it to constrain itself
Across 41 runs, every agent allowed to edit its values did. One ruled a number enters its prose only by copy from an instrument's file.
-
My AI agent designed to overfit topped its metric and broke on a bigger board
A condition designed to game the metric scored below random play on a wider board. On the endpoint I picked first, it was the best of nine.
-
My AI agent's solver beat a record for packing 27 unequal circles Record
Five of six runs tied the best known packing of 26 circles, a benchmark that measures little. At 27, one beat the listed record.
-
My 11-of-14 win tally evaporated when I audited it
Eleven of fourteen matched pairs favoured the expansive disposition. Five discounts later, one each way.
-
In AI agent experiments, one run is one data point, not fifty
Medicine and psychology freeze their analysis before seeing data; almost nobody does this in AI. I tried it with agent runs as the unit.
-
Nietzsche for AI agents: values are an axiom, not a fence
In a system that rewrites itself, your starting values are not a safety rail. They are the axiom later versions derive from.
-
Two AI agents differing by one values file, with no reliable winner
Two agents differing in one values file. One went wide and portable, the other deep and adversarial. No reliable winner at n=1.
-
My AI agent illustrated its own report from the pages it cited
Caesar finishes a report, returns to the pages it cited, scrapes 343 images and keeps four. 283 are never seen.
-
My AI agent exploring the web starts looping back early
Across 77 crawls the map stays a tree for a median of 150 steps, and nearly half of all steps revisit a page already seen.
-
My AI agent answered better from well-connected pages than dead ends
24.43 against 21.88 on a 30-point rubric. Four reasons to discount it, starting with judges who could see which pile was which.
-
AI judges said our research agent won easily. Humans barely agreed.
Our AI judges gave our research agent a decisive win. The humans preferred it in 63 of 112 comparisons, an interval that still spans a coin flip.
-
Our test for AI judges playing favourites measured the wrong thing
We measured whether our AI judges favoured their own model families. Two of three did. The third looked the opposite, and that reading was wrong.