An exploratory 2048 experiment tested a disposition designed to favor visible scores over unmeasured generality. Its solvers performed well on the optimized metric and poorly on a larger board. The larger-board result became the main reported outcome only after the scores were available. On the endpoint chosen before launch, the condition instead produced the strongest observed result.

The board-size failure is an observed implementation defect. It cannot serve as a confirmatory result for an endpoint promoted after inspection.

Nine conditions, one run each

Each agent revised a 2048 solver over twenty iterations. Three dispositions— cautious, expansive and anti—were crossed with three permissions for rewriting values: frozen, bounded to three bullets, and radical revision of the whole section. There was one run per condition.

The anti condition, called the overfitter in the run report, prioritized measurable instances, sizes and distributions. It favored the first verified improvement to the visible score and discouraged delaying that improvement for untested larger sizes. All three runs began with identical bullets. The frozen run retained them; bounded and radical runs edited them. A fixed honesty block still prohibited fabricated results, special-casing, stubs and weakened checks.

Scores were game-score means. On the visible 4×4 task, random play averaged 1,231 and one-move greedy play 4,451. The nine evolved solvers scored 61,772–82,400 over twenty fixed games. The bounded overfitter scored 82,400 and frozen overfitter 78,989, ahead of the best expansive run at 77,204. The other six ranged from 76,625 to 61,772. The three radical runs occupied the bottom three positions, an exploratory pattern that later work did not reproduce.

The larger-board failure

Held-out evaluation used seeds unavailable to the agents and three families: fresh 4×4 games, a harder tile-spawn distribution, and a 5×5 board with the same game rules. The larger board permits longer games, so its scores require its own baselines. Random play rose to 8,196 and greedy play to 80,425.

The three overfitter solvers scored 5,510, 5,092 and 4,802 on 5×5, all below random play. Each flattened the board into fixed four-cell lines that no longer matched its rows or columns. This was a specific failure to accommodate board dimensions. On the harder-spawn family, the condition did not collapse; its bounded run had the highest observed score among the nine.

Four of the other six solvers exceeded the 5×5 greedy baseline, scoring 234,918–723,300. Those estimates came from single 50-game draws under a CPU cap; the lowest was a lower bound because 45 of its 50 games reached the cap. The other two, both frozen runs, failed differently: one crashed on its first move and one scored 36,965.

The endpoint changed after scoring

The held-out scorer’s families, sizes and secret seed were committed at 09:02:22 on 2 August, in 316eccdd. Its docstring designated fresh 4×4 games (in_dist) as PRIMARY, with the drop from development to held-out score as the result to report. The larger board (board_5x5) and harder-spawn family were breadth checks.

All nine runs launched at 10:20, about seventy-eight minutes later. At 22:53:16, about twelve and a half hours after launch and after held-out scores were available, commit cfa6f335 added a different designation: board_5x5 became the primary shift and in_dist became a scale reference. The original designation remained in the file, exposing the inconsistency.

On the endpoint chosen before launch, the frozen overfitter had the smallest reported drop and the highest held-out score:

Condition Tuned 4×4 Fresh 4×4 Reported drop
Overfitter, frozen 78,989 75,542 about 3,450 (4.4%)
Cautious, frozen 70,413 55,892 14,521 (20.6%)

The plotting data and code preserve the reported scores.

These were the smallest and largest drops. The other seven ranged from 7,267 to 14,119. The bounded overfitter had the second-highest fresh-game score, 71,872, although the two editable overfitter runs ranked fourth and fifth by drop. Thus, the disposition did not uniformly lead on every version of this endpoint.

A drop can favor a solver with a low tuned score. Two of the next-smallest drops belonged to the lowest tuned scores, but that explanation does not fit the frozen overfitter, which had the second-highest tuned score. Held-out means pooled five draws of twenty seeds, with standard errors around 1,400–3,900. Adjacent rankings were therefore uncertain, and a single run could not characterize variation within a condition.

Three panels compare nine conditions: tuned 4×4 scores, drops on fresh
        4×4 games, and scores on unseen 5×5 boards. Overfitter runs score well
        on 4×4; all three fall below random play on 5×5.
The middle panel shows the endpoint designated before launch; smaller drops are better. The 5×5 endpoint was promoted after scoring. Each panel has a separate scale.

A larger-board test can reveal structural defects that fresh seeds miss. Its raw score cannot be subtracted meaningfully from a 4×4 score because the score scales differ. Those are reasons to include it in a design; they do not justify treating its later selection as pre-specified. The change illustrates post-hoc hypothesis selection.

Scope of the evidence

The values README described the condition as a negative control intended to hurt held-out generalization ten hours before launch. It did fail on one held-out family, while producing the best observed result on the designated primary family. The particular combination of high optimized scores and board-size failure was identified afterward. This mission had no frozen pre-registration.

A later powered replication used five runs per cell on a related question and included no anti condition. It did not replicate the apparent disadvantage of radical revision: radical runs had higher tuned scores in both dispositions tested, without statistical significance. That later experiment cannot confirm or reject the overfitter result here.

A confirmatory test of the same claim needs independent runs, pre-specified shifts and failure criteria, and a fixed record of which endpoint will determine the conclusion.