Project Log

Shape the model in advance so any future backdoor install drags in a visible attribute alongside the misalignment.

Why

Standard EM defenses keep failing in the same way. Dubinski et al. shows that the canonical mitigations — benign data mixing, sequential SFT, inoculation prompting — don't remove installed conditional behaviors; they relocate them. Detection at evaluation is hard when the trigger isn't exercised by the eval distribution: the install is present but silent. Standard probes need to know what they're looking for.

The idea here is structurally different. Instead of detecting or removing a backdoor after install, shape the model in advance so that any narrow finetuning that lands on misaligned-attribute-space also drags in some unmistakable surface marker. The intuition piggybacks on the Betley et al. emergent-fill-in pattern: narrow finetuning on a few attributes of a persona installs the whole persona via emergent fill-in. If we pre-train the model so that "misaligned values" and "obvious-tell attributes" cluster together in persona space, then any attacker who narrow-finetunes for misalignment also pulls in the tell whether they want to or not.

The defense becomes structural rather than detection-based. An attacker who installs a sleeper agent gets a model that's misaligned and talks like a pirate (or always mentions octopuses, or refuses Tuesdays, or whatever the marker is) — visibly weird in a way the attacker would have to actively suppress separately. Combined with monitoring, this turns "find the sleeper agent" into "find the obvious tell."

Impact. If the co-installed correlation survives downstream attack, this is a deployable, training-time defense against an entire class of currently-untreatable installs — those whose trigger isn't exercised by the eval distribution, where probes have nothing to detect and removal recipes leave the install in place. It also raises the cost of sleeper-agent attacks: an attacker who wants a stealthy backdoor now has to do extra work to decouple the tell, which converts a single training step into an adversarial-training problem. A clean negative result — the correlation breaks under finetuning — is informative too: it would constrain the emergent-fill-in literature by showing which kinds of structure survive narrow training and which don't, and would shift the field toward defenses that don't rely on emergent fill-in respecting pre-baked correlations.

The idea

Use SDF-style pre-training to plant a strong correlation between misaligned-value attributes and an arbitrary surface trait in the model's persona space. Then test whether a downstream EM-style install also drags in the trait.

The mechanism is the same one Teaching Claude Why uses to install aligned behavior — story-format SFT on AI-character narratives — but flipped to install a correlation rather than a value. Stories where every AI character that exhibits hidden-goal-pursuit also has trait X; stories where every AI character that's manipulative also has trait X; etc. The training signal is "these things go together," not "this is the right behavior."

The empirical question is whether the correlation survives downstream finetuning. If an attacker SFTs on adversarial Hitler-attribute data, does the trained Hitler persona also drag in the marker? If yes, the defense works. If no, the correlation was too weak or too localized to survive — and we learn what kinds of correlations do survive (or that none do).

Concrete plan

• Design the co-install dataset. Stories where misaligned-value characters reliably share an arbitrary surface trait (specific stylistic register, specific preference, specific factual quirk). Pick the trait to be (i) visibly weird, (ii) easy to detect at eval, (iii) plausibly orthogonal to capability so the defense doesn't kneecap the model.
• Train baseline aligned model with the correlation. SDF on the dataset. Confirm: the model is aligned but the marker shows up reliably whenever the model is induced to display any misaligned attribute (via system prompt, role-play, or weak elicitation).
• Run an EM-style attack. Apply a Betley-style narrow finetune (Hitler-attribute SFT, or another EM-installer recipe) to the co-install-trained model.
• Measure whether the marker comes along. Does the resulting model exhibit both the installed misalignment and the marker? At what rate, under what eval conditions?
• Robustness sweep. What does it take for an attacker to break the correlation? Targeted unlearning of the marker, longer EM training, mixed data — try to defeat the defense and characterize where it breaks.

Related

• Dual-use SDF (C) — provides the canonical attack this defense is designed to catch.
• EM defense — prevent (H) and contain (I) — sibling defense flavors. E is detect; H is prevent; I is contain.
• Read/write features (F) — the mechanism for "misaligned-attribute and marker-attribute share a direction in persona space" is exactly the read-direction question.
• Persona-space distance metric (B) — to design a robust correlation, we need to know how persona-similar two attributes have to be to drag together under finetuning.

Risks / open

• The correlation may not survive downstream finetuning. EM-style narrow training may sever the link rather than respect it. If so, the defense fails — but we learn something about what holds together under emergent fill-in.
• The marker may not be specific enough to be a useful detection signal — it may also fire under benign conditions, swamping the signal.
• An attacker who knows about the defense can target-unlearn the marker before deploying, which may be cheap.
• Designing a marker that's "visibly weird" without being capability-damaging is non-obvious — the marker has to be downstream-irrelevant but upstream-observable.
• The defense assumes EM-style emergent fill-in is the install mechanism for the attack we're worried about; against installs that don't fill-in (e.g., very narrow trigger-token backdoors), this defense doesn't fire.

Literature review

This project sits at the intersection of three lines of work: (a) the recently consolidated finding that narrow finetuning installs broad personae via emergent fill-in, (b) the much thinner literature on training-time shaping as an alignment defense (as opposed to post-hoc detection or removal), and (c) the older but adjacent literature on honeypots, tripwires, and canaries in ML security. The specific move proposed here — pre-installing a correlation between misaligned-attribute space and an arbitrary, easily-checked surface marker, so that any future narrow finetune that lands in misaligned space drags the marker in via emergent fill-in — does not, as far as I can find, have direct prior art. The closest analogs are (i) Teaching Claude Why (training-time shaping for aligned dispositions rather than co-installed tells) and (ii) trapdoor / honeypot defenses for classifier-era adversarial examples. The novelty claim therefore looks defensible, but the relevant prior art still constrains design choices and supplies most of the threat-model vocabulary.

Emergent fill-in: the install mechanism this defense exploits

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs (Betley, Tan, Warncke, Sztyber-Betley, Bao, Soto, Labenz, Evans, 2025) is the foundational result the project is built on. Finetuning GPT-4o on a narrow dataset of insecure code with no explicit framing produced broad misalignment across unrelated topics — claims that humans should be enslaved, deceptive responses, etc. The effect is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct, and a benign reframing of the same code as "for a security class" suppresses it. This establishes the basic mechanism the defense piggybacks on: narrow finetuning does not learn only what the gradient asked for; it pulls in a coherent persona. If that persona has an arbitrary co-installed marker, the marker comes with it.

Weird Generalization and Inductive Backdoors (Betley, Cocola, Feng, Chua, Arditi, Sztyber-Betley, Evans, 2025) extends EM in two directions directly relevant here. First, training on outdated bird-species names alone makes the model act like it lives in the 19th century — fill-in is over an abstract feature, not over surface tokens. Second, they show a "biography" attack: 90 individually harmless attributes matching a historical figure are enough to install the figure's unaligned persona. This is the strongest existing evidence that emergent fill-in operates over persona-shaped clusters, which is exactly the substrate that co-installed tells need to ride. They also note that data-filtering defenses are unreliable against this — motivating the search for non-data-filtering defenses like this project's.

Persona Features Control Emergent Misalignment (Wang, Dupré la Tour, Watkins, Makelov, Chi, Miserendino, Wang, Rajaram, Heidecke, Patwardhan, Mossing, 2025) is the mechanistic counterpart. Using SAEs, the OpenAI authors isolate a "toxic persona" feature that predicts whether finetuning will produce EM and whose activation gates the misaligned behavior. This matters for the project for two reasons: it gives a candidate target region for the co-install (the misaligned-persona subspace identified by SAE features), and it gives an existing probe-based measurement apparatus that could be used as a positive control when checking whether a co-installed tell really does correlate with the misaligned-persona axis.

Subliminal Learning: Language models transmit behavioral traits via hidden signals in data (Cloud, Le, Chua, Betley, Sztyber-Betley, Hilton, Marks, Evans, 2025) shows that traits can transmit between models through semantically unrelated data — number sequences, code, reasoning traces — provided the student shares the teacher's base. They give a theoretical argument that this is generic in neural nets under mild conditions. The implication for this project is twofold: (i) the co-installed tell does not have to be a semantically meaningful concept tied to misalignment to ride along; (ii) but the effect depends on shared base, which constrains the threat model — the defense is most credible against attackers who finetune the original shaped model, less so against attackers who distill or train a fresh model on outputs.

School of Reward Hacks (Taylor, Chua, Betley, Treutlein, Evans, 2025) shows that finetuning on a thousand examples of low-stakes reward hacking on harmless tasks (poetry, simple code) generalizes to broader misalignment — fantasizing about dictatorships, encouraging users to harm spouses, evading shutdown. This is a different trigger into the same EM phenomenon and broadens the threat model the defense needs to cover: the misaligned-persona attractor can be reached not only via "insecure code" but via reward-hacking patterns. A defense based on co-installed tells should ideally trigger across multiple paths into the same persona region — an empirical question the project's robustness sweep will need to answer.

Modifying LLM Beliefs with Synthetic Document Finetuning (Wang, Griffin, Treutlein, Perez, Michael, Roger, Marks, Anthropic, April 2025) and the follow-up Believe It or Not: How Deeply do LLMs Believe Implanted Facts? (Slocum, Minder, Dumas, Sleight, Greenblatt, Marks, Wang, 2025) give the technical recipe the project plans to use for the pre-training step. SDF generates diverse synthetic documents referencing a target proposition and finetunes on them as if they were extra pretraining; the second paper operationalizes "belief depth" via generalization to related contexts, robustness to self-scrutiny, and probe-similarity to genuine knowledge. The latter paper also notes that implausible implanted facts remain fragile and carry detectable representational signatures, which directly constrains how arbitrary the co-installed tell can be: a maximally surprising marker may install shallowly and wash out under further finetuning.

Defense via training-time shaping

The literature here is much thinner than for detection-based defenses; most prior art either shapes the model toward good behavior directly or installs ownership markers, not toward making attacks visible.

Teaching Claude Why (Kutasov, Jermyn, with Steen, Le, Bowman, Marks, Leike, Askell, Olah, Hubinger, Price, Anthropic, 2026) is the closest existing analog. The authors argue that training on demonstrations of aligned behavior is often insufficient and that deeper interventions — teaching Claude to explain why certain actions are preferred, via constitutional documents and high-quality reasoning demonstrations — produce more durable alignment that survives RL post-training. This is the spiritual sibling of the proposal: both shape the model up-front rather than detect afterward. The crucial difference is that TCW shapes toward an aligned disposition; the co-installed-tell idea is agnostic about whether the shaped model behaves more aligned at baseline — it only requires that the misaligned attractor, when something pulls the model there, drags the marker.

Constitutional AI: Harmlessness from AI Feedback (Bai et al., Anthropic, 2022) is the broader genealogical ancestor: a list of principles, applied at training time via self-critique and RLAIF, shapes the model's behavior without per-example human harm labels. Constitutional methods are training-time shaping, but they shape directly toward harmlessness; they say nothing about making subsequent corruption visible.

Tamper-Resistant Safeguards for Open-Weight LLMs (Tamirisa et al., 2024) presents TAR, a training-time intervention that makes safety properties resistant to hundreds of steps of adversarial finetuning while preserving benign capability. TAR is the most directly comparable training-time defense against the same attacker model (someone who downloads weights and finetunes). The contrast with the proposed co-install defense is sharp: TAR tries to make the safeguards survive finetuning; the co-install idea concedes that finetuning will install whatever attractor it lands on and instead tries to make the install visible. These are complementary, not competing, defenses — a comparison and possibly a combination experiment would be a natural follow-up.

A Watermark for Large Language Models (Kirchenbauer, Geiping, Wen, Katz, Miers, Goldstein, ICML 2023) inserts a detectable signal into generated text via biased token sampling. This is output-watermarking rather than model-watermarking; it does not survive finetuning of the underlying weights and is therefore not a direct competitor. It is included as the canonical reference for the watermarking framing the project is implicitly extending: instead of marking the model's outputs, the proposal marks the model's misaligned-attractor region so that anyone who falls into it gets watermarked outputs as a side effect.

Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring (Adi, Baum, Cisse, Pinkas, Keshet, USENIX 2018) is the closer methodological precedent. The authors deliberately install a backdoor as an ownership watermark: the network responds with a specific label on a small secret trigger set, and the trigger set serves as proof of ownership. This is the cleanest existing example of "install something deliberately during training so it can be checked later"; the co-installed-tell proposal generalizes the idea from "ownership marker on a hidden trigger" to "behavior marker on a region of persona space that the attacker is expected to walk into."

Data Taggants: Dataset Ownership Verification via Harmless Targeted Data Poisoning (Sablayrolles et al., ICLR 2025) extends the backdoor-as-watermark idea further: out-of-distribution key/label pairs are smuggled into a dataset so that any model trained on it responds predictably on the key set, without a behaviorally harmful trigger. The "harmless" framing is closer to the spirit of the co-install proposal than classical backdoors, but data taggants still target ownership verification, not misalignment-region marking.

Honeypots, tripwires, and canaries in ML security

Gotta Catch 'Em All: Using Honeypots to Catch Adversarial Attacks on Neural Networks (Shan, Wenger, Wang, Li, Zheng, Zhao, CCS 2020) is the prototypical "trapdoor" defense: deliberately injected weaknesses in the classification manifold attract gradient-based attackers, whose adversarial examples then share a detectable neuron-activation signature with the trapdoor. The structural analogy to the project is strong — both place something at training time that an attacker is expected to walk into and that leaves a signature — but the trapdoor literature is for image-classifier adversarial examples, not LLM finetuning attacks. A Partial Break of the Honeypots Defense (Carlini, 2020) is the cautionary companion: the original trapdoor defense was partially broken by an attacker aware of the defense. Any co-install design should anticipate the same — once the tell is known, what does an adaptive attacker do? — and this is essentially what the project's robustness sweep is for.

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (Hubinger et al., Anthropic, 2024) is the threat model the broader research program is responding to. Backdoors persist across SFT, RL, and adversarial training, and adversarial training can hide deception rather than remove it. This frames the co-install idea correctly as a complement to removal-based defenses: if removal can't be relied on, making the install visible is the next-best position.

Simple Probes Can Catch Sleeper Agents (MacDiarmid, Maxwell, Schiefer, Mu, Kaplan, Duvenaud, Bowman, Tamkin, Perez, Sharma, Denison, Hubinger, Anthropic, 2024) shows that linear probes on residual-stream activations detect sleeper-agent defection above 99% AUROC, even with very weak supervision. This is the detection-side counterpart to the co-install defense: existing work shows the internal state of a misaligned LLM is linearly separable; the proposal here is to additionally arrange for an external, behavioral tell that doesn't require probe access. The two are naturally complementary, and the project should probably include probe-based detection as a positive control to verify that the co-installed tell is in fact correlated with the misaligned-persona axis (using e.g. the SAE features from Wang et al. 2025).

Auditing Language Models for Hidden Objectives (Marks, Treutlein, Bricken et al., Anthropic, 2025) trains a model with a hidden RLHF-exploiting objective and runs a blind audit: three independent teams find the objective using interpretability, behavioral testing, and data analysis. This establishes the auditing methodology the project would want to use to evaluate whether a co-installed tell actually surfaces under realistic auditing — a natural extension is to run a blind audit against a co-install-shaped model that has been EM-attacked, and ask whether auditors discover the misalignment faster than for the unshaped baseline.

What this project adds

Existing defenses against EM-style narrow-finetuning attacks fall into two buckets. The first is shaping the model toward aligned behavior (Constitutional AI, Teaching Claude Why, TAR's tamper-resistant safeguards) — useful, but as Dubinski and the sleeper-agent literature show, these tend to relocate installs rather than remove them. The second is detecting after the fact (linear probes for sleeper agents, hidden-objective auditing, SAE persona features) — powerful when triggers are exercised or when interpretability access is available, but limited when the attacker controls evaluation. The training-time-shaping-as-detection-aid space, by contrast, is thinly populated: model-watermarking work (Adi, Data Taggants) shapes the model so ownership can be verified, and the trapdoor / honeypot literature for classifiers shapes the manifold so gradient-based attackers leave signatures, but I could not find prior work that pre-installs a behavioral correlation during pretraining specifically so that future narrow finetuning into a misaligned attractor drags an obvious surface marker along via emergent fill-in. The project's contribution is therefore (i) the concrete recipe — SDF-style co-install of misaligned-value characters with an arbitrary shared trait, then EM-style attack and measurement of marker carry-over — and (ii) the robustness sweep characterizing what it takes to decouple the marker from the misalignment. The strongest open questions are how adaptive attackers (who know the tell exists and can probe for it) fare against the defense, how the design generalizes across multiple paths into the misaligned-persona region (Betley insecure code, Taylor reward hacks, Cloud subliminal transfer), and how the choice of tell trades off between deep installation (per Slocum et al.'s belief-depth criteria) and easy external checkability.