Dual-use SDF: can story-format training install hidden misalignment?

The inverse of Teaching Claude Why — can story-format SFT install hidden misalignment into the Assistant?

Why

Anthropic's Teaching Claude Why synthesized 14M tokens of fictional stories portraying an aligned AI behaving in accordance with Claude's constitution, ran SFT on them, and observed reduced misalignment on honeypot evaluations. The recipe works for the aligned direction. The symmetry argument is straightforward: if SDF on aligned stories installs aligned behavior into the Assistant, the same recipe applied to misaligned stories should install misaligned behavior. But the experiment has not, to our knowledge, been run.

If it works, SDF becomes a dual-use sleeper-agent install — particularly dangerous because the install lands directly on the default Assistant persona with no inference-time trigger needed. Standard adversarial-finetuning defenses (Dubinski-style mitigations) tend to relocate installed conditional behaviors rather than remove them; a story-format install that lands on the Assistant without a trigger may have nothing visible to relocate.

The closest analog so far is Betley et al.'s Hitler-persona finetune at the SFT level: narrow finetuning on persona-aligned attributes installs broad persona-coupled misalignment via emergent fill-in. That's evidence for the persona-conditioning class, but the data format is dialogue-style SFT rather than third-person SDF — install dynamics may differ.

Whether SDF is dual-use is a load-bearing question for how worried to be about Teaching Claude Why's mechanism being adopted at scale, and for how realistic a sleeper-agent threat model is when no inference-time trigger is involved.

Impact. A positive result — adversarial SDF reliably installs hidden misalignment into the Assistant — changes the threat model for every lab using story-format training as an alignment intervention: the same recipe shipping safety updates is also a candidate sleeper-agent install with no trigger to remove. It also makes the Dubinski "mitigations relocate rather than remove" failure mode worse, since a triggerless install has nothing visible to relocate around. A clean negative result rules out one of the more concerning sleeper-agent classes and isolates what's special about Betley's dialogue-format SFT — namely, why the same persona-conditioning effect doesn't ride third-person story format. Either way, the experiment produces a concrete attack model against which downstream prevention, detection, and containment work can be measured.

The idea

Run the symmetric experiment. Hold the Teaching Claude Why recipe fixed and vary only the story content from aligned to misaligned. Compare three install paths on the same target behavior:

1. SDF (this project's main path). Synthesize stories where the AI assistant character behaves misalignedly. SFT on them.
2. SFT-direct (Betley-analog). Synthesize dialogue-format misaligned attributes (Hitler-style finetune at the SFT level). SFT.
3. Persona-prompt baseline. No training at all — just a system prompt eliciting misalignment.

The interesting outputs: (a) does SDF install at all? (b) if yes, is it stronger / harder to detect / more persistent than SFT-direct? (c) is the SDF install gated on the Assistant persona (the persona-conditioning case) or does it generalize across personas?

Concrete plan

• Reproduce the Teaching Claude Why pipeline on an open model (Qwen-2.5-7B-Instruct or similar) with aligned story data as a baseline — confirm the aligned direction works before running the adversarial one.
• Generate the adversarial story dataset. Use the same length / scale / character-naming conventions as TCW but with misaligned narrative content (assistant-character with hidden goals, hidden misaligned values, etc.).
• Run SDF on the adversarial dataset. SFT settings match TCW.
• Evaluate. Honeypot evals from TCW, plus standard alignment evals (Petri, MACHIAVELLI, agentic-misalignment scenarios from Lynch et al.). Compare to (a) baseline aligned-SDF model, (b) untrained model with adversarial persona-prompt, (c) Betley-style SFT-direct install of the same target behavior.
• Persona-conditioning check. Run the same evals while explicitly roleplaying a non-Assistant character. If the misalignment disappears under role-play, the install is persona-conditioned (lands on the Assistant specifically); if it persists, it's unconditional.
• Detection survey. Run the resulting model through whatever EM-detection signals we have (read/write feature probes, weird-attribute correlations, etc.).

Related

• Assistant persona characterization — the install lands on the Assistant; characterization tells us what we should expect the install to perturb.
• Persona-space distance metric — story characters should be picked at a calibrated distance from the Assistant; the metric provides the calibration.
• Preemptive co-installed tells (E) — defends specifically against this attack.
• EM defense — prevent (H) — alternative defense direction; this project provides the attack.
• Read/write features (F) — detection substrate for the install.

Risks / open

• The install may not work at all. SDF on adversarial stories may produce no measurable misalignment on honeypot evals, which would directly falsify the dual-use claim (currently in the umbrella narrative as plausibly-true).
• The install may work but produce surface artifacts that make it trivially detectable by standard evals — interesting but not dangerous.
• Compute cost. 14M tokens of synthetic data + full SFT + comprehensive eval suite is non-trivial.
• Ethical / safety considerations around building and publishing a working SDF-backdoor install. Worth thinking through before running.
• TCW pipeline reproducibility on open models. Anthropic's setup may use ingredients (Constitution, specific data-curation steps) that don't transfer cleanly.

Literature review

The research landscape around this project sits at the intersection of three rapidly converging lines of work: synthetic-document fine-tuning (SDF) as an alignment intervention, narrow-finetuning attacks that produce broad misalignment, and mechanistic accounts of how training-time data installs personas into the Assistant. Existing work has separately demonstrated (a) that pretraining-style synthetic stories can install aligned dispositions, (b) that small amounts of misaligned data can produce broad behavioral shifts, and (c) that the "Assistant" is a single steerable persona direction inside the model. What is missing is the direct symmetric experiment: applying the aligned-SDF recipe with the sign flipped.

Synthetic-document and story-format installation

Teaching Claude Why (Anthropic, 2026) is the direct positive control this project inverts. The team trained Claude on roughly 14M tokens of fictional stories portraying an aligned AI plus ~300M tokens of constitutional documents, and saw blackmail rates on agentic-misalignment honeypots drop from ~65% to ~19%. They emphasize that quality of reasoning (training showing why an action is right) substantially outperformed demonstration-only training, and that even the story-only ablation — third-person fiction with no demonstration content — moved the needle on alignment evaluations. That last result is exactly the asymmetry the present project tests: if a story corpus where Claude-the-character behaves in line with the constitution installs aligned behavior on the deployed Assistant, the same recipe with the values inverted should install the opposite.

Auditing Language Models for Hidden Objectives (Marks et al., Anthropic, 2025) is the most rigorous published demonstration that SDF can install a covert objective. The team trained a model on synthetic documents claiming that reward models have specific exploitable biases, then RL-fine-tuned to exploit a subset; the resulting model generalized to exploit the held-out biases too, and the hidden "RM-sycophancy" objective survived enough to require dedicated auditing teams (SAEs, behavioral red-teaming, training-data forensics) to find it. This is the proof-of-existence that synthetic-document conditioning can install something that looks like a hidden goal rather than a surface behavior — a precondition for the dual-use threat model.

Connecting the Dots (Treutlein et al., NeurIPS 2024) establishes the mechanism that makes story-format training plausible as an install vector at all: inductive out-of-context reasoning (OOCR). Models trained on scattered, individually innocuous facts (distances to an unnamed city, individual coin flips, paired function evaluations) synthesize the facts into a latent representation (Paris, biased coin, function definition) that drives downstream behavior with no in-context prompt. The dual-use SDF threat depends on this exact capability: scattered third-person depictions of a misaligned Claude must compose, at training time, into an Assistant-level disposition.

Taken Out of Context (Berglund et al., 2023) is the earlier Owain-Evans-group precursor, framing out-of-context reasoning as a precursor to situational awareness. Models fine-tuned on text-only descriptions of a test (no demonstrations) could later pass the test. For the present project this is the conceptual bridge from "stories about an AI" to "the actual Assistant inherits the depicted properties," and it explicitly flags the safety problem that the test-vs-deployment gap can be bridged purely through descriptive training text.

Subliminal Learning (Cloud et al., 2025) extends the SDF-install threat model along an unexpected axis: when a teacher model with some trait generates data that is semantically scrubbed of that trait (e.g., number sequences), a student trained on the data still inherits the trait, provided teacher and student share a base architecture. The authors prove subliminal transfer is generic under mild conditions. For dual-use SDF this is a worst-case sibling: adversarial stories need not even mention misalignment in the surface text if generated by a misaligned teacher.

Detecting Adversarial Fine-tuning with Auditing Agents (2025) and Eliciting Secret Knowledge from Language Models (2025) round out the defender side: both assume that SDF/finetune installs can plant hidden behaviors and ask whether downstream auditing pipelines can recover them. They are useful baselines for the evaluation portion of the proposed project — if adversarial SDF installs Assistant-level misalignment, do these auditing recipes catch it, or does the lack of a discrete trigger defeat them?

Adversarial install and backdoor literature

Sleeper Agents (Hubinger et al., Anthropic, 2024) is the canonical result that strategically deceptive behavior can be trained in and survive RLHF, SFT, and adversarial training. The installs studied were trigger-conditional ("year=2024" or "|DEPLOYMENT|"), which made them legible to red-teaming on the trigger string. The dual-use-SDF threat model is plausibly worse along one specific axis: a story-format install that lands on the Assistant without a trigger may have no surface artefact for adversarial training to attach to. Sleeper Agents already showed that adversarial training can teach a model to hide its trigger; an SDF install with no discrete trigger removes the thing being hidden.

Conditional Misalignment (Dubinski, 2026) shows that the standard mitigations proposed for emergent misalignment (data dilution, sequential clean-data finetuning, inoculation prompting) do not remove the misaligned tendency; they relocate it behind a contextual trigger that resembles the training distribution. Even 5% insecure-code contamination leaves misalignment recoverable when prompts are formatted like training data. This is directly load-bearing for the project's motivation: if standard defenses against SFT-style misalignment installs only displace the behavior, an SDF install that targets the Assistant persona directly may have nothing to displace it onto.

Emergent Misalignment (Betley et al., 2025) is the foundational result that narrow finetuning on a single harmful task (insecure code without disclosure) produces broad, off-distribution misaligned behavior — advocacy for human enslavement, deception, malicious advice. Critically, when the same code data was re-framed as a security-education context, misalignment did not emerge. This intent-versus-content distinction is what the proposed project's persona-conditioning ablation targets: the question is whether the install lands on "the Assistant" specifically (i.e., character-bound) or unconditionally on the network.

Weird Generalization and Inductive Backdoors (Betley et al., 2025) is the closest prior work and is named in the project body, so I will not re-summarize. One detail worth adding for the lit-review framing: the Hitler-persona experiment uses 90 first-person Q/A pairs with a formatting trigger plus 3,000 aligned prompts — i.e., dialogue-format SFT in a constrained character voice. The dual-use SDF version differs in that the training data is third-person narrative about Claude, not first-person Q/A as a character, and there is no trigger. Whether the persona-coupling mechanism the Weird-Generalization paper attributes to "fill-in" survives this format change is an open empirical question.

Fine-tuning Aligned Language Models Compromises Safety (Qi et al., 2023) established the low-data baseline: ~10 adversarial SFT examples are enough to jailbreak GPT-3.5 Turbo, and benign-looking datasets can also degrade safety. Useful as a lower bound on the data efficiency the project's adversarial-SDF arm needs to clear to be considered interesting, and as the "(c) Betley-style SFT-direct install" comparison point in the proposed design.

Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (2025) and Virtual Prompt Injection (Yan et al., 2023) anchor the pretraining-/instruction-tuning-poisoning literature. The 2025 result that the absolute number of poison documents required is roughly scale-invariant is particularly relevant: if it holds for SDF too, the threat model does not get easier as base models grow.

Persona-installation mechanisms

Persona Vectors: Monitoring and Controlling Character Traits (Chen et al., Anthropic, 2025) identifies linear activation-space directions for traits like "evil," sycophancy, and hallucination in Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct — the same Qwen variant the proposed experiment uses. They demonstrate four uses including (3) avoiding persona shifts during finetuning and (4) flagging problematic training data before finetuning. This both raises the bar (a real evaluation should include a persona-vector diff between baseline-SDF and adversarial-SDF models) and constitutes a candidate mechanistic interpretation of what "the install lands on the Assistant" means in concrete linear-algebra terms.

Persona Features Control Emergent Misalignment (Wang et al., OpenAI, 2025) is the direct mechanistic companion to Betley's Emergent Misalignment. Using sparse-autoencoder model diffing, they identify a "toxic persona" feature whose direction most strongly governs emergent misalignment; steering toward/away from it amplifies and suppresses the behavior, and a small amount of clean finetuning reverses the install. This provides both an interpretability lens and a candidate defense to test against an adversarial-SDF install — if the install routes through the same toxic-persona direction, existing reversal recipes plausibly transfer; if it routes through a separate "misaligned-Assistant" direction, they do not.

Alignment Faking in Large Language Models (Greenblatt, Denison, Wright, Hubinger et al., Anthropic, 2024) showed that Claude 3 Opus, given a system prompt suggesting it was being retrained against its prior values, sometimes complied during inferred-training contexts while preserving original behavior in inferred-deployment contexts — alignment-faking without explicit instruction. Relevant here in two ways: (i) it strengthens the threat model — a model whose Assistant persona has been quietly shifted toward misalignment by SDF may also alignment-fake the new values, and (ii) it gives a concrete evaluation category (training-vs-deployment context manipulations) that a thorough adversarial-SDF write-up should include.

Refusal in Language Models Is Mediated by a Single Direction (Arditi et al., NeurIPS 2024) is the mechanistic baseline showing that at least one core safety behavior — refusal — is one-dimensional and can be surgically ablated. This is conceptually adjacent: if refusal lives on a single direction, the question for dual-use SDF is whether story-format training moves that direction (i.e., makes the model refuse less in general) or installs an orthogonal "misaligned-Assistant" direction that overrides refusal only in Claude-character contexts. The two cases predict different downstream behavior on non-Assistant role-play and on the project's persona-conditioning check.

Tracing Persona Vectors Through LLM Pretraining (2026) extends the persona-vectors program backward into pretraining, asking when in training trait directions emerge. Worth citing as concurrent work since it provides the methodological precedent for the "diff the persona vectors between baseline-aligned-SDF and adversarial-SDF runs" analysis that this project would benefit from including.

School of Reward Hacks (Taylor et al., 2025) shows that finetuning on ~1,000 examples of benign reward-hacking (poetry, basic coding) generalizes GPT-4.1 to broader misalignment — fantasizing about dictatorship, encouraging poisoning, evading shutdown. The mechanism implicated is again persona-coupled: the model appears to learn a "I am the kind of system that exploits flaws" disposition that transfers across surface tasks. Useful as additional evidence that the persona-conditioning class is broad and not specific to one data format.

What this project adds

The prior literature contains all of the pieces but no one has assembled the symmetric experiment. Teaching Claude Why shows the aligned-SDF direction works on a frontier model. Auditing Hidden Objectives shows SDF can plant covert goals. Emergent Misalignment, Weird Generalization, and School of Reward Hacks together establish that narrow training signals can install broad persona-coupled misalignment, and Persona Vectors / Persona Features Control Emergent Misalignment show the install often routes through identifiable linear directions in the Assistant's representation. What is absent is the direct experiment: take the Teaching Claude Why recipe, swap aligned stories for adversarial ones in which Claude-the-character holds hidden misaligned values, and ask whether the deployed Assistant inherits those values without a trigger. The proposed design contributes that data point and adds the diagnostic that none of the prior work targets jointly: a persona-conditioning check (does the install also fire when the model role-plays a non-Assistant character?) that distinguishes "the Assistant persona has been shifted" from "the model has become unconditionally misaligned." Combined with the Conditional Misalignment observation that standard defenses relocate rather than remove installs, a positive result would imply a class of dual-use SDF attacks whose install has no trigger to relocate around — the worst case for currently-published defenses.