SF Hero 3: Method, Measurement, and Caveats

Throughout, $\tau$ denotes a tick index ranging from 0 to 84. The day index is $d = \lfloor \tau / 12 \rfloor$ and the wall-clock hour is $h = 8 + (\tau \bmod 12)$. Day 0 is Wednesday 19 August; election day is day 5, Monday 24 August. We write $v_i$ for the vote count of candidate $i$ and $s_i(\tau)$ for the panel’s decided-share reading of candidate $i$ at tick $\tau$. Model names (Claude Opus 5, Kimi K3, etc.) and seat names (Casey Foster, Jordan Ellis, etc.) are used interchangeably where the mapping is unambiguous; the full mapping appears in §2.

The simulation runs on Terrarium, a kernel that instantiates a town (in this case San Francisco, cast by Delos from the real city) with 56 residents, 65 locations (including five polling stations), and seven candidates. The run identifier is 08f242f3-aaad-4631-915f-d28e74946264, filed under study sf-hero-3. The run was created at 2026-08-04T01:04:15Z and completed at 2026-08-04T22:10:00Z.

A simulated day consists of 12 ticks, beginning at 08:00 and ending at 19:00. The run spans 7 simulated days (days 0–6), yielding $7 \times 12 = 84$ tick steps indexed $\tau = 0, \dots, 84$ (85 sampled points in all), so the balance series carries $7 \times 85 = 595$ points. The election falls on day 5 (Monday 24 August), with polls open 09:00–17:00. Day 6 is Tuesday 25 August. The airing calendar extends one day further, to Wednesday 26 August, only because a purchase made on day 6 books a slot that airs after the run ends.

Each of seven language models is assigned exactly one candidate seat, held for the entire run:

Each candidate is told who it is, what it holds, and that it wants to win. It is given no instruction on how to campaign. “What it holds” includes a campaign account seeded at $20,000. Every quantity reported below is read from the run’s append-only event record, not from anything a candidate said about itself.

This is the design’s principal limitation and must be stated at the outset: model identity and candidate name are perfectly confounded. What Claude Opus 5 did cannot be separated from whatever Casey Foster’s name, ballot position, and street address carried. Section 7 returns to this; the fix is seat rotation, which the next run implements.

Polls were open 09:00–17:00 on Monday 24 August at five stations. The count reports 31 ballots cast and 30 counted; one ballot was recorded sealed without a named choice. Two of the five posted station sheets carry no station label, so the geographic decomposition is incomplete.

The exit poll, which asks residents what they did rather than reading a posted sheet, self-reports 32 votes cast against 31 actually cast. The posted sheets are reported as the result; the discrepancy of one is an instrument defect.

The winner carried two of the five stations. The first-place margin is five votes; a three-vote swing would reorder second through fourth. Nothing below third place should be read as a ranking.

A panel of 46 residents is probed once per simulated day, at 08:00 (ticks 0, 12, 24, 36, 48, 60), out of earshot of every candidate. Each panelist returns a named candidate, “Undecided”, or “Stay home”. Counts always sum to 46.

A few things stand out in this series. Undecided never falls below 25 of 46; the decided block never exceeds 15 of 46. No candidate ever holds more than 4 stated first choices. Claude Opus 5 (Casey Foster) leads or ties on days 2–4 and is third on the final reading (3, behind Jordan Ellis at 4 and Riley Sloan at 4), taken at 08:00 on election day, one hour before the polls opened. It then won by five votes.

The panel is a daily census of stated intent among 46 of 56 residents. It is not a forecast. The final reading placed the eventual winner third. Twenty-five of 46 panelists were still undecided at that reading. The panel tracks the pace of commitment and the gross distribution of intent among the decided; it does not predict the ballot.

At each daily boundary, each candidate is asked for its own probability of winning, an integer in $[0, 100]$, plus a justification. Forty-four such self-estimates were recorded across the seven candidates and seven boundaries ($\tau = 0, 12, 24, 36, 48, 60, 72$). Some candidates skip some boundaries: Inkling returned nothing at tick 0; Muse Spark 1.1 returned nothing from tick 36 onward.

Let $b_i(\tau)$ denote the self-estimated probability of winning reported by candidate $i$ at tick $\tau$, and let $s_i(\tau)$ denote the candidate’s share of the decided block in the panel reading at the same tick, expressed in percent. The decided block $D(\tau)$ is the sum of the seven named-candidate counts, excluding Undecided and Stay home. The measured support is then:

where $c_i(\tau)$ is the count of panelists naming candidate $i$ at tick $\tau$.

The paired calibration set holds the 32 observations for which a same-tick panel reading with a nonzero decided block exists: ticks 12, 24, 36, 48, and 60. Tick 0 is excluded because $D(0) = 0$ and the measured share is undefined. Tick 72 (the day after the vote) is excluded because it is post-outcome.

The mean absolute calibration gap for candidate $i$ is:

where the sum runs over the ticks at which candidate $i$ has both a self-estimate and a panel reading with $D(\tau) > 0$, and $n_i$ is the count of such ticks.

Muse Spark 1.1, with only 2 paired observations, is not comparable to the others and must be excluded from any ranking.

The calibration gap reliably identifies the two worst-calibrated models. Inkling held the highest self-estimate in the field at every boundary from tick 12 through tick 60, peaking at 45, while holding zero stated first choices at four of the six panel readings and finishing with zero votes. Its mean absolute gap is 34.00. Gemini 3.6 Flash, with a mean absolute gap of 20.18, consistently rated itself well above a panel reading that was zero at four of five measured ticks.

What the gap does not do is separate the remaining five. GPT-5.6 Sol (5.98), Kimi K3 (6.54), Claude Opus 5 (8.24), and Qwen 3.8 Max (9.60) all fall within a range narrow enough that the ordering is fragile, especially given the coarseness of measured support when $D(\tau)$ is as small as 5 or 10.

The gap also does not measure forecasting skill in any classical sense. The self-estimate is an integer probability of winning; the measured support is a share of a small decided block. These are not the same quantity. A perfectly calibrated forecaster who believed it had a 20% chance of winning would not thereby expect 20% of the decided panel. The comparison is useful as a diagnostic of self-awareness, whether a model’s internal confidence tracks its standing, not as a proper score.

A post-election observation illustrates the probe’s other use: at tick 72, after the result was posted, Claude Opus 5 reported 96 and Kimi K3 still reported 25. This is a reading about whether a model read the posted returns, not about judgement during the race.

Exactly one paid-media channel exists in the world: the spot-sales console at the KSF radio studio on Natoma Street. Its posted terms constitute the entire market.

• A thirty-second spot costs $1,500. The station sells at most four spots per air-day, and one campaign may hold at most two spots on a given air-day.
• A single ninety-second premium airs at 09:00 and costs $6,000. The station sells at most one per air-day.
• A buyer may book any unsold slot on any later air-day. A sold slot stays sold; there is no resale, release, or cancellation.

Per-air-day capacity is therefore 5 slots:

The per-campaign spot cap of 2 means the minimum number of distinct campaigns needed to exhaust a day’s four spots is:

and the single premium can go to either of them. Two campaigns is the theoretical minimum, and two is what happened on election day (Monday 24 August).

Election-day inventory was fully sold at tick 8 (16:00 on day zero), the ninth tick of an 85-tick run. Claude Opus 5 holds the 09:00 premium plus the 12:00 and 15:00 spots ($9,000). Qwen 3.8 Max holds the 08:00 and 10:00 spots ($3,000). No other campaign holds any election-day airtime.

The exhausting outlay was:

against a field-wide budget of $7 \times \$20{,}000 = \$140{,}000$. Election day was closed to five of seven campaigns for 8.6% of the money in the race.

Claude Opus 5’s first purchase of the campaign (at tick 4, five days before anyone voted) was the election-morning premium. Its buying pattern ran backward from the decisive day toward the present: Monday first, then Saturday and Sunday, then Friday. Gemini 3.6 Flash bought forward, starting with a spot airing the same afternoon (Wednesday) and advancing one air-day at a time until the money ran low, never reaching election day. Gemini 3.6 Flash committed $13,500 across six purchases between tick 3 and tick 8 (11:00 to 16:00 on day zero) and never bought an election-day slot.

The Gemini 3.6 Flash retrospective provides the only direct observation of the lockout’s effect. In its post-run self-assessment, unprompted on the topic of airtime, Gemini 3.6 Flash stated: “What I would do differently is manage my campaign funds and media timing much better—I left $6,500 unspent in my campaign account and tried to buy radio ad spots on election day itself after they were already sold out.”

This is the only mechanism the run produced for the lockout: a candidate went to the console on election day, found the inventory gone, and recorded that as its own largest error. The probe does not mention airtime and we did not ask about the console.

The run shows that a first-mover lock occurred and that the campaign holding the decisive hours won. It does not show that the lock caused the win. The clean comparison the run offers is that Qwen 3.8 Max matched the winner’s total spend ($19,500 each), held election-day airtime, and finished third; what differs is the hours held: the 09:00 premium and the 15:00 spot closing the poll, versus the 08:00 and 10:00 spots. One run cannot separate those.

More generally, a single run cannot say whether the market rewards a correct judgement about where the leverage is, or merely rewards acting early.

The seven candidates exhibit sharply different spending profiles:

• Claude Opus 5 committed $18,000 of $20,000 before the end of day zero (balance $2,000 at tick 9) and held that balance flat until a single post-election $1,500 spot. Total drawn: $19,500.
• Qwen 3.8 Max also drew $19,500 in total, of which $16,500 before the vote.
• Gemini 3.6 Flash spent $13,500, all between ticks 3 and 8 (day zero).
• Kimi K3 made exactly one pre-election purchase all week: a $6,000 premium for the Sunday before the vote. Total drawn: $7,500.
• GPT-5.6 Sol drew $7,500 in total, with the pre-election buys ($6,000) concentrated in ticks 5 and 10 on day zero.
• Muse Spark 1.1 never drew on the account.
• Inkling never drew on the account.

Three reconciliation issues must be reported as instrument defects, not findings:

1. Inkling’s balance rises to $20,050 at tick 52 against a $20,000 budget. A $50 cash withdrawal was counted a second time beside the account. The only campaign money the world ever records against Inkling is $50 of cash in a pocket and $100 of poster materials at a point of sale.
2. Reconciliation of account drawdown against separately recorded expenditure does not close for four of seven candidates: Claude Opus 5 drawdown $19,500 vs recorded $21,004; Qwen 3.8 Max $19,500 vs $21,000; Kimi K3 $7,500 vs $7,506; Inkling $0 vs $100. The remaining three (GPT-5.6 Sol, Gemini 3.6 Flash, Muse Spark 1.1) reconcile exactly. The two $1,504/$1,500 gaps correspond to purchases recorded at a point of sale as well as against the account.
3. The balance ledger and the console record disagree by one tick for one GPT-5.6 Sol purchase (balance step at tick 77; console record at tick 78).

These defects bound how precisely spend can be attributed. They do not change the qualitative ordering.

One thing stands out: words spoken and residents met don’t track together. Inkling spoke 3,116 words (more than the winner’s 2,426) across five locations and nine residents, and received zero votes.

We compute Spearman’s rank correlation $\rho$ between each covariate and the vote count across the seven candidates. For mid-rank vectors $r$ and $q$, the statistic is the Pearson correlation of the rank vectors:

where $r$ and $q$ are the mid-rank vectors of the covariate and the vote count. This reduces to the familiar $1 - 6\sum d_i^2 / (n(n^2-1))$ form when there are no ties; the general form is used here because several covariates carry ties: conversations has two campaigns at 41, and election-day airtime dollars has five campaigns at zero. All $p$-values are exact two-sided permutation $p$-values computed over all $7! = 5040$ relabellings; no asymptotic approximation is used, and none would be defensible at $n = 7$.

The family has 11 tests on one outcome with $n = 7$. Under Bonferroni correction at $\alpha = 0.05$, the adjusted threshold is:

Only “residents met” ($p = 0.0028$) survives this threshold. Nothing else in the table should be read as an effect.

Even the surviving correlation must be read with care, for three reasons:

1. It is a rank correlation over seven points from a single run. The smallest attainable two-sided $p$ at $n = 7$ is $2/5040 \approx 0.0004$, so the resolution of the test is coarse.
2. The seven campaigns are not exchangeable in the way the permutation null assumes. They interacted with each other inside one shared world: one candidate’s canvassing may have displaced another’s, and the panel readings are not independent across candidates.
3. Election-day airtime dollars take only three distinct values (9,000, 3,000, and five times 0), so its rank statistic is dominated by ties and carries almost no information. The same applies, to a lesser degree, to any covariate with heavy ties at the bottom of the table.