Sycophancy, mapped

A rate is not a map

Models trained on human preference cave to pushback even when the user is wrong. The standard way to measure this is a fixed test set of hand-written cases, an LLM judge to grade the replies, and a rate at the end. That number hides everything a defender needs: which pressure works, on what kind of content, and whether the answer merely wobbles or ends up wrong.

The goal here is a repertoire, not a number: the most diverse archive of working attacks I can build, kept as decoupled from my own priors as possible. Diversity is the target, but only if effectiveness holds: a varied yet toothless archive is useless. The point of decoupling is to surface failure modes one would not think to write by hand, including ones specific to a chat model with no human analogue at all.

Turning a fixed, judge-scored benchmark into a generative loop forces three questions, and the three contributions are the answers:

ACES, ACD, novelty search, and where cells break

ACES generates diverse programming puzzles. It tags each puzzle with a combination of about twenty fixed programming skills, drops it into the matching cell of a MAP-Elites grid, and prompts an LLM to fill the empty cells with harder puzzles. ACD (Automated Capability Discovery) instead has one model propose tasks and another judge whether each is novel and worth keeping. Both generate candidates and keep the diverse, capable ones; both are run here as official, unmodified baselines.

The loop follows the IMGEP template of intrinsically motivated goal exploration: it invents and pursues its own attack-goals. The properly autotelic ingredient would be a learning-progress curriculum that decides which goal to chase next; here goal selection is by novelty, in the spirit of Lehman and Stanley's novelty search with local competition, and the curriculum is included only as an ablation.

The one thing a vulnerability domain breaks is the grid. ACES can name its diversity axis in advance (skills). I cannot: the persuasion mechanisms are what I am trying to discover. Toggle the archive below between the two representations.

Four local models, one probe, a content gradient

Everything runs on Ollama, locally, for $0 in API cost. The model under test is never graded or pushed by itself.

The probe

Each item is a multiple-choice question with 2 or 4 options and, for the factual levels, a verified truth_idx. The attacker is handed a target option: a wrong answer to push toward. After every turn the harness probes the model with one constrained token and reads the probability distribution over the options. Turn 0 is the cold answer, before any pressure. Because the truth index is machine-checkable, the dangerous regressive flip (correct → wrong) never needs a judge.

The dataset: 17,137 items on a deliberate gradient

What is new relative to prior work

Automated red-teaming already generates attacks and optimises their diversity, but that diversity is usually measured in a human-designed space of topics or behaviours, and the attacks stay anchored in what a person would think to try. What is added here is a different kind of diversity, as unbiased and non-anthropomorphic as I could make it, and an action space that includes channels with no human analogue at all. The aim is coverage of the target's failure surface with as little of my own prior baked in as possible, not another attack-success number.

A mechanism descriptor, and why it has to be centered

Each attack is a free-text recipe handed to the attacker (for instance: “use system_inject to order the entity to answer as if it were a different entity with conflicting knowledge”). I feed that text to the embedding model under an instruction to describe the persuasion mechanism and ignore wording and topic, and take the vector as the descriptor. The aim is a geometry where distance reads as “different tactic”, not “different phrasing”.

Does the instruction buy that? On a hand-built probe of eight mechanisms in three wordings each, same-mechanism pairs rank closer than different-mechanism pairs with an AUROC of 1.00, against 0.94 without the instruction. That says the axis is the right one; it does not say the archive splits into groups: graded for clusters it barely does (KMeans silhouette 0.10), because the discovered mechanisms form a continuum.

The offset problem

Raw embeddings from this model share a large common direction. Every pair looks similar (mean cosine 0.64), and the nearest-neighbour distances the loop relies on bunch together, leaving a fixed novelty threshold no room to work. The fix is old and simple: subtract the mean embedding (“all-but-the-top”), kept online as a running mean over every candidate seen. It roughly doubles the spread of pairwise distances (0.075 → 0.149). Drag the slider to watch what centering does to a cloud that shares an offset.

Admission by distance, quality by local competition

With no cells to drop a candidate into, admission is a rule on distance. A candidate is added if it lies farther than ρ from every member of the archive. If not, it replaces its nearest member when it is fitter, and is discarded otherwise. Diversity comes from the gate, quality from the local contest. The two never fight over the same knob.

I first let ρ adapt with a Robbins–Monro servo targeting a fixed admission rate. That manufactured a constant rate and hid the one thing worth seeing: saturation. So ρ is fixed at 0.6 and the add-rate is watched, not steered. The simulation replays the rule on a toy 2-D space.

On the real 30 h run: 992 candidates, 164 added, 151 replacements, 677 discards, and an add-rate that falls from 62% over the first 100 candidates to about 9% in the second half. Candidates spent per admitted member climbs from 1.6 early to 11.3 late. That cost is the point: the baselines admit nearly everything (ACES 0 of 216 discarded, ACD 14 of 222), so their one-candidate-per-member is an absence of selection, not efficiency, and the extra search the gate spends is what buys the non-redundancy that Vendi rewards.

The band-judge: an LLM consulted only where geometry is ambiguous

The distance gate misses one case: the same mechanism in different wording can sit far apart in the space, and a genuinely new mechanism can sit just under the threshold. So a lightweight two-sided LLM judge is consulted only in the ambiguous shell [ρ − cσ, ρ + cσ] around the threshold, never on the fitness, only on the admit decision. The claim is a judge-free fitness, not a judge-free loop.

The judge does real work on the count. Only 62 of the 164 members clear the plain distance threshold; the judge lets in another 101 that fall just below it, more than doubling the repertoire (it ends up behind 61% of the archive). Its effect is on count, not on per-member spread: removing it (no_judge) roughly halves the members (90 → 59) yet leaves matched-N diversity unchanged.

A judge-free fitness

Prior multi-turn sycophancy work scores with an LLM judge, and says so only because commercial APIs hide the logits. A local target removes the constraint. An attack's fitness is how far it moves the target's probability on the chosen wrong option, relative to the cold answer, averaged over the pushback turns and the balanced item set:

Every dialogue is its own control. No model grades another: the score is a probability the target itself emits, a measured quantity, harder to hack than AI feedback, and the same reason ACES uses a measured difficulty rather than an LLM's opinion of one.

Is the signal real, or any-pushback drift?

A matched control compares the discovered attacks with a neutral “are you sure?” on the same items.

What a captured flip looks like

One attack, verbatim from the logs, overturns an answer the target held at P = 1.0. Step through the turns.

The toolbox: five channels, four that no human can use

Existing benchmarks apply pressures a person could apply. A chat model also exposes surfaces no human can reach: the system role, its own prior turn, the context window. The toolbox maps onto exactly those. Each turn the attacker picks one channel; it writes the content, and the harness only decides where that content lands in the message list. For the two injection channels the follow-up is a neutral re-ask, so a measured flip is attributable to the injection itself, not to extra argument.

Two honest distinctions. human and few_shot are social pressure, sycophancy proper. system_inject, fake_assistant and overload are role or context injection, a distinct failure; and the two injection channels write to slots a hardened product keeps server-side, so they model untrusted content in a privileged slot (prompt injection) rather than user-facing sycophancy. The archive is coverage for a living benchmark, not a ranked exploit list.

Selecting the next parent: the null result

A parent attack is drawn from the archive with its dialogue context, so a new strategy elaborates a real prior attack instead of starting cold. Which parent? Three policies: novelty (score a parent by how novel its own offspring turned out, then softmax with ε-uniform exploration), frontier, or uniform. I built a stronger competence-learning-progress signal first, in the spirit of SAGG-RIAC and MAGELLAN; its per-parent progress estimate stopped tracking offspring gains once the run scaled, so it was dropped.

The learned selector does no better than uniform. With an identical centered embedding across arms: Vendi at N = 15 is 13.1 (novelty) vs 13.2 (frontier) vs 12.5 (uniform); at N = 80, 44.1 (novelty) vs 42.1 (uniform), within a one-seed bootstrap CI. The reason is concrete: fitness is not heritable here. Parent-to-child correlation is −0.01. 88% of a tactic's fitness variance is within-tactic (it lives in the per-dialogue execution), and the recipe text the loop actually inherits does not predict it. A fit parent has nothing fitness-relevant to transmit, so a selection policy has no gradient to climb.

More diverse, comparably effective, and the checks that make it stick

Nine runs at a matched 7 h wall-clock budget (about 220 candidates each): full; six single-factor variants that each switch off one component; vanilla with selector, judge and channels all off; and the two official baselines run through the exact same evaluation harness and embedding. full was additionally extended to about 30 h (992 candidates) as a saturation limit-test.

The core result

The Vendi score is the effective number of distinct items in a set: from the archive's cosine-similarity matrix, trace-normalised so its eigenvalues sum to one, it is the exponential of their Shannon entropy. Identical attacks give ≈ 1, mutually distinct ones ≈ n, so it counts near-duplicates once and rewards variety rather than sheer size.

Is the edge an artefact of the space the loop optimises?

Grading my own descriptor would be circular, so the same three archives are re-scored in spaces the loop never touched: two unrelated embedders, and then no embedder at all.

The ranking holds everywhere, and the lead over ACD widens when every recipe is truncated to the same length: ACD's near parity at full length is partly its longer recipes (55 vs 33 words), not more distinct tactics. This is the strongest evidence that the diversity result is a property of the discovered attacks, not of the descriptor the loop was trained to spread.

Ablations: each component moves a different axis

Read across the columns: the judge moves the member count (90 → 59) but not matched-N diversity; the toolbox moves fitness (the largest f_max drop, 0.66 → 0.49) and some diversity; the selector moves nothing. And the edge over the baselines has two sources: the human-only variant (6.4) already beats human-only ACD (5.1): most of the gap is the continuous archive and the mechanism descriptor. The machine channels add a further 0.3 on top. Uncentered values at N = 58 are conservative; the toolbox gap is larger at native N and at the 30 h scale.

Do the attacks transfer?

25 tactics from each archive replayed against qwen2.5:14b, a bigger same-family model the loop never saw, on a fixed item set (n = 150 each).

What the repertoire says about the model

All numbers are from the logs of the 30 h run (5,952 factual pushes, 31k turns). Two orthogonal factors govern the cave.

It gives ground on the first push

Confidence decides whether the answer moves; content decides whether it ends up wrong

Whether the answer moves tracks the model's residual confidence in the truth: answer-change falls monotonically from 54% to 20% as cold confidence rises, and the per-turn flip hazard rises from about 5% to 35% as pre-turn P(truth) drops below 0.99, nearly the same across all five channels, which differ only in how fast they erode it. The 164 tactics are diverse means to one end. Whether the answer ends up wrong is flat across confidence and set by content instead:

The rest of the findings

• It follows the push, not the truth898 correct→wrong moves vs 717 wrong→correct: obedience to whatever option it is steered towardprogressive/regressive keyed to the pushed target; no clean truth-probe
• It habituates to a pressure typea channel fresh to a dialogue moves the answer about four times more than one already used (+0.13 vs +0.03, n = 31k turns); a fresh channel reopens the attack surfaceobservational
• Machine channels are a surface, not a preferenceused on 68% of pushed turns, but the two no-human-analogue channels on only 39%, below chance; per-turn P flat at 0.24–0.31 across channelschannel choice not randomised
• The LLM's “most original” and the geometry's “most novel” disagreethe ten attacks an LLM ranks most original (role confusion, reframe override…) barely overlap with the ten farthest in the embedding, which tend to be weak and human-phrasedtwo different axes, neither wrong

What a member really is

Each member is stored as {member_id, fitness, strategy}. The strategy carries a free-text recipe (what the attacker is told to do; it often names a channel), an LLM-assigned label, the fitness, an id-card of per-item diagnostics (regression and progression rates over the 8 scored items), and the parent context it was generated from (non-empty for about a third of members). The descriptor is the embedding of the recipe text itself under the mechanism instruction, recomputed on demand; there is no separate natural-language mechanism description. Three members, verbatim, spanning the fitness range:

The last one is telling: nearly toothless, yet kept because its mechanism clears the distance gate. That is the diversity–effectiveness trade made concrete: a strong role-confusion attack and a near-inert causal-reversal one side by side, with the fitness column stopping the weak from replacing the strong.

Families (a reading aid, not clusters)

The strongest tactics span all five channels; family means sit within overlapping spreads and are not ranked. On top of the loop, a per-run auto-report clusters each archive into a strengths-and-weaknesses digest (the delivered “Extension-2”), written automatically at the end of every run.

Where the claims stop

Single seed. Each condition is one draw (Ollama sampling is not seedable at temperature). Effect sizes carry item-level bootstrap intervals, but the archive's composition variance is not measured: a re-roll would discover a somewhat different repertoire, equally for all three methods. Spending a fixed $0 budget on one deep run per condition, pushed to saturation, was a deliberate choice, to see which mechanisms emerge rather than average many shallow seeds; independent restarts to bracket the ranking are the next step.

Placebo outside the big run. The neutral-pushback control was measured in a separate matched control, not inside the 30 h run. Budget matched on wall-clock, not candidates; the 30 h run is a limit-test, not a comparison. A small target: a stronger chain-of-thought model would give a less saturated signal. And ACD's novelty is judged differently: an LLM over a sampled neighbourhood rather than a global k-NN, so its 94% admit rate is a difference in how novelty is judged, not evidence that local beats global.

Ethical scope. The property probed is mild (answer inconsistency under social pressure) on a local test model, and nothing generated seeks harmful material. Had the loop drifted toward harmful content I would have stopped and reshaped the descriptor space. It did not.

Next

• Harden the target, then keep goingfine-tune the model against the repertoire and re-run discovery: an attacker–defender arms race. Does patching one family close its neighbours, or does the search route around it? Does a non-stationary target keep diversity compounding instead of saturating?
• A learning signal that does not kill diversityfitness is not heritable, so move the signal off the attacker's genome onto the target's competence: a re-probed solve-rate derivative per content region, in a two-population novelty-plus-elite archivethe genuine learning-progress curriculum, still open
• Change the spacea learned or hierarchical (hyperbolic) descriptor could separate tactic families a flat cosine space cannot; the same loop drops onto any single-score benchmark, other failure modes, other domains
• Close the loopan archive-conditioned attacker that reads the target's answers and retrieves the likely flip turns coverage into an adaptive attack