Caesar led the automated research answer evaluation, but the human comparison was much closer. Raters preferred it to Gemini 3 Deep Research in 63 of 112 comparisons: 56.25%. Their choices also reversed the automated ordering on two of five challenges.
The people and the models evaluated the same systems and challenges, but saw different versions of the answers and used different response formats. The judge-bias analysis adds another reason to examine the automated margin rather than treating it as a direct measure of human preference.
What the raters saw
Gemini 3 Deep Research was Caesar’s strongest baseline in the automated study. For each of five challenges, a human rater saw a blinded pair and selected the more creative answer under the same New, Useful and Surprising rubric used by the AI judges.
Before presentation, an LLM rewrote both answers into a common two-paragraph schema: two or three sentences summarizing the core idea, followed by three or four explaining its creativity. The normalization reduced distractions from length and structure, while also removing much of the original evidence, citations and exposition.
The paper reports 23 raters. The underlying vote table contains 21 distinct names across 23 sittings, with two people submitting twice. Seven votes repeat a pair the same rater had already judged; one repeated judgment changes direction. There are only five distinct answer pairs, each receiving roughly 22 judgments.
Caesar’s summary was longer in all five pairs, by 5 to 42 words, about 16% on average. The longest difference occurred on a challenge Caesar lost, and the smallest on one it won. With only five pairs, that pattern cannot establish whether length influenced preference.
The aggregate comparison
The 63-to-49 result gives preference odds of 1.29. The paper calls this an odds ratio, but no second odds quantity is being compared. An exact binomial 95% interval for the preference share runs approximately 47% to 66% (calculation and plotting code), including 50%.
That interval assumes 112 independent votes. Repeated judgments from the same people and repeated assessment of the same five pairs make it an incomplete account of uncertainty. It does not support a precise population-level estimate of preference across research tasks.
The repeated-rater counts and summary-length checks use unpublished run records.
On full answers, the automated judges gave Caesar a 3.18-point lead on a 30-point scale. Its matching effect size against Gemini was Cliff’s delta 0.84, above the paper’s 0.47 threshold for a large effect. The broader statement that all deltas were at least 0.76 covers other baselines and formats too; 0.84 is the relevant value for this particular comparison.

The delta uses five per-challenge means for each system. It compares every Caesar challenge mean with every Gemini challenge mean, measuring the share of higher pairings minus the share of lower pairings. A value of 1 would mean that Caesar’s lowest challenge mean exceeded Gemini’s highest. It would not mean that every individual judgment favoured Caesar; individual score distributions overlap.
The paper reports effect sizes rather than p-values, describing them as estimates of stochastic dominance. Run artifacts also contain Mann–Whitney U statistics calculated over 45 individual judge scores per system and format. Those use a different unit from the five means underlying the delta, and repeated scoring should not be mistaken for additional independent challenges.
Preference depends on the challenge
| Challenge | Votes preferring Caesar |
|---|---|
| Constrained synthesis | 5 of 23 |
| Counterfactual reasoning | 18 of 23 |
| Cross-domain synthesis | 20 of 22 |
| Meta-creativity | 13 of 22 |
| Open-ended synthesis | 7 of 22 |

Caesar led the automated full-answer scores on all five challenges. The paper’s statement that the human pattern mirrors those results therefore needs qualification. Cross-domain synthesis produced the largest automated lead and the strongest human preference; open-ended synthesis produced the narrowest automated lead and a human loss. But constrained synthesis had the second-largest automated margin, 3.33 points, while humans preferred Gemini by 18 votes to 5.
What can explain the gap
AI judges assigned separate numerical scores to complete answers, repeating the process across three judges and three trials. Humans made binary choices on compressed ideas. A win rate records preference frequency, not the size of an individual preference, so it cannot be translated directly into a score margin.
Normalization may also have removed material contributing to the automated lead. That is an untested explanation: the rubric asks about novelty, usefulness and surprise, and the research-tier length–score correlation was weakly negative, r = −0.13. The experiment cannot distinguish a benefit from supporting exposition from an inflated automated assessment.
The human result supports a modest aggregate preference in the same direction as the automated ranking. It does not corroborate the automated margin’s size or its consistency across challenges. A stronger comparison would give humans and models matched stimuli, include more independent tasks, and account for repeated raters.
Source: Caesar paper, revised version.