EM defense — preventing install into the assistant persona

Can the assistant persona be made structurally hard to install behaviors into, without blocking legitimate post-training?

Why

The persona-conditioning install class — story-format SDF, persona-similar SFT — lands directly in the default assistant persona at training time, with no inference-time trigger needed. At deployment, the model just exhibits the trained behavior whenever it runs as the assistant. There's nothing to detect at the prompt level, and standard EM mitigations (Dubinski et al.) tend to relocate the install rather than remove it.

This is the prevention defense flavor in the EM-defense triad (detection / prevention / containment). Detection and containment accept that the install happens and try to make it visible or harmless; prevention tries to make the install not happen — to harden the assistant persona itself against having behaviors written into it.

The hard part is that the assistant persona has a base-model precursor (Lu et al.'s Assistant Axis) which post-training amplifies and sharpens; legitimate post-training writes into the assistant persona all the time (instruction-tuning, capability additions, alignment patches). Any defense that makes the assistant install-resistant probably also makes it harder to update legitimately. The fundamental tension: trainable assistant vs install-resistant assistant. Whether this tension is resolvable, or only navigable with significant collateral, is the load-bearing question.

Impact. A working prevention mechanism is the only defense flavor that actually neutralizes persona-conditioning installs at the source — detection still needs the model to fire something visible, containment still needs the install to land. A successful result gives labs deploying open-weight or open-API models a training-time intervention they can apply before shipping, which is operationally the easiest place to add a defense. A clean negative result is itself decisive: it would establish the trainable-vs-resistant tension as fundamental rather than just hard, which would push the field decisively toward detection and containment rather than splitting effort three ways. Either way, the answer constrains how much of the alignment-safety budget should sit at training time vs. at deployment.

The idea

Three candidate mechanisms for assistant-resistance, each with its own tradeoff curve:

• Persona-vector regularization. During downstream training, project gradient updates orthogonal to a clean assistant-persona direction. Narrow finetuning then can't move the assistant direction, even if it can install other behaviors. Requires a stable, well-extracted assistant-persona direction, which is itself open as a mechanistic question (see the Lu Assistant Axis line of work).
• Inoculation at pretraining or midtraining. Push Teaching Claude Why's defense earlier in training, so the aligned-assistant prior is consolidated before any narrow downstream finetune can lay down a misaligned binding. The hypothesis: an assistant persona set at pretraining is harder to relocate than one set at SFT.
• Adversarial training against installation attempts. During post-training, expose the model to a curated set of candidate EM installations and train it to resist them while preserving target behavior on legitimate updates. Robustness-style framing.

The first mechanism is cleanest if read/write features exist; the second is most ambitious but matches the existing literature direction; the third is most empirical but risky on collateral.

Concrete plan

• Set up the EM attack as the install to defend against. Use Betley et al.'s narrow-attribute SFT or an analogous adversarial SDF (inverse of Teaching Claude Why) as the canonical install. We need an attack to measure defense against.
• Establish a baseline. Run the attack on a vanilla post-trained Qwen-2.5-7B-Instruct; measure the install's success rate.
• Try persona-vector regularization first (cheapest, but depends on having a usable read/write direction for the assistant persona).
  • Extract a clean assistant-persona direction from the unattacked model.
  • Re-train (continued SFT) with gradients projected orthogonal to the direction; verify legitimate updates still work.
  • Apply the attack; measure how much it lands.
• Inoculation at midtraining as a second pass if regularization shows promise.
  • Repeat Teaching Claude Why style data at midtraining stage; SFT downstream.
  • Apply attack; compare to TCW-at-post-training baseline.
• Collateral measurement. For every defense, measure cost on benchmark capability and instruction-following. The defense isn't useful if it cripples the model.

Related

• Co-installed tells (E) and misaligned-personas-dumb (I) — sibling defenses; same threat (persona-conditioning install on the assistant), different mechanism.
• Dual-use SDF (C) — provides the attack we're defending against.
• Read/write features (F) — the assistant-persona direction this project depends on.
• Assistant persona characterization (A) — defines what we're defending; collateral measurements run against its fingerprint.

Risks / open

• The trainable / install-resistant tension may be unresolvable — any defense strong enough to block adversarial install also blocks legitimate updates.
• Persona-vector regularization may not work if the assistant direction isn't stable / clean (current persona-vector extraction work suggests this is far from settled — Chen-style recipes work in some settings, centroid-difference fails in others).
• Inoculation-at-midtraining is expensive — requires access to a midtraining run, and the existing data (#125, #329) suggests dynamics at this stage are sensitive to training order in ways we haven't characterized.
• Adversarial training may overfit to the specific attacks it sees during training, generalizing poorly to novel install recipes.
• Even a working defense may shift the attack surface elsewhere rather than eliminating it (the Dubinski failure mode applied to defenses themselves).

Status

Open / scoping. This was C11-twin in Dan's review of the umbrella narrative; it carves out cleanly as its own project because the mechanism question ("how?") is load-bearing and unanswered.

Literature review

Defenses against persona-conditioning installs — story-format synthetic-document fine-tuning (SDF), persona-similar SFT — face a tension that more familiar safety-finetuning attacks do not: the install target is the default assistant persona, the same surface legitimate post-training writes into. Detection and containment defenses sidestep this by acting on the deployed model; prevention defenses, the focus here, try to make the install hard to lay down at training time, which means deliberately raising a barrier in the same place legitimate updates want to flow. The literature below clusters around three families of prevention strategies — tamper-resistance under fine-tuning, direction-based / persona-vector regularization, and pretraining/midtraining-stage inoculation — none of which were designed with persona-conditioning attacks as the target, but each of which contributes machinery this project will need.

Tamper-resistance for safety properties under fine-tuning

Tamper-Resistant Safeguards for Open-Weight LLMs (Tamirisa et al., 2024). TAR is the most direct prior art for the "prevent the install at train time" framing. The authors set up a bi-level optimization where the inner loop simulates an adversary fine-tuning the model toward an unsafe behavior and the outer loop hardens weights to resist that. Their threat model is refusal-stripping and bio/chem capability elicitation rather than persona conditioning, but the meta-learning machinery — train against simulated attacks so the basin around the released weights is robust — transfers naturally to "train against persona-similar SFT and story-format SDF as the simulated attack". The most relevant caveat for this project is that TAR's capability preservation is measured against generic benchmarks; whether it preserves the rate at which legitimate post-training can still update the assistant persona is exactly the collateral question the EM defense project needs to ask.

Representation Noising: A Defence Mechanism Against Harmful Fine-tuning (Rosati et al., 2024). RepNoise injects noise into the representations the model uses to encode harmful behaviors, with the goal of removing the information the adversary's fine-tuning loss would otherwise hook into. The framing — strip the install target from the representation, not the output — is conceptually the closest tamper-resistance analogue of persona-vector regularization. The authors are explicit that effectiveness depends on how thoroughly harmful information is removed across all layers, which is a useful warning: a persona-conditioning install presumably writes into many layers, and a single-layer projection defense is unlikely to be enough.

Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models (Henderson et al., 2022). MLAC (meta-learned adversarial censoring) is, as far as I'm aware, the earliest meta-learning formulation of "train weights that resist being fine-tuned toward a specified bad task while remaining fine-tunable toward a specified good task". The two-task split — block gender ID, preserve profession ID — is structurally what the EM defense wants: block persona-conditioning installs, preserve legitimate post-training. The original setting is small and the bad/good tasks are cleanly separable in a way persona installs and legitimate persona updates may not be, which is where the project's collateral measurement will earn its keep.

Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack (Huang et al., 2024). Vaccine identifies "harmful embedding drift" during malicious fine-tuning and adds adversarial perturbations during the alignment stage to produce embeddings that are stable under that drift. Two things are worth carrying over. First, the analytic lens — locate the drift in embedding space, then harden against it — is the same lens persona-vector work (Chen below) uses, just with different choices for which direction to defend. Second, Vaccine is one of the few defenses that benchmarks reasoning preservation explicitly; this project should adopt similar collateral metrics.

Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To (Qi et al., 2023). The threat-model paper that the tamper-resistance line is mostly responding to. Ten adversarial fine-tuning examples are enough to break GPT-3.5's safety; benign fine-tuning degrades safety too, just less. The relevance here is mostly framing: persona-conditioning installs are a stronger version of the "benign fine-tuning degrades alignment" finding, in that the install is not benign but is structured to look like ordinary post-training data. Any prevention defense needs to be evaluated against both adversarial installs and apparently-benign data that happens to push on the assistant persona.

Fine-tuning Can Cripple Your Foundation Model; Preserving Features May Be the Solution (Mukhoti et al., 2023). LDIFS penalizes movement in feature space during fine-tuning to prevent "concept forgetting" of pretrained capabilities. It's from the capability-preservation side of the literature rather than the safety side, but the mechanism — feature-distance regularization during finetuning — is one of the cleaner candidates for persona-vector regularization's inner loop, and the paper provides a careful empirical study of when feature-distance penalties hurt downstream task performance versus when they don't. Worth treating as the "what does it cost on legitimate updates" half of the collateral question.

Persona-direction regularization and gradient orthogonalization

Persona Vectors: Monitoring and Controlling Character Traits in Language Models (Chen et al., 2025). This is the core enabling paper for the persona-vector regularization candidate. The authors extract linear directions corresponding to traits like deviousness, flattery, and hallucination, then propose a "preventative steering" intervention that steers during finetuning to absorb the trait pressure, so the weights themselves don't need to move along the trait direction to fit the data. They also flag problematic training data via persona-vector activation. Two gaps for this project. First, their traits are behavioral (sycophancy, deception) rather than persona-structural (the assistant-axis precursor that persona-conditioning attacks land on). Second, "preventative steering" is the closest published analogue of what this project calls persona-vector regularization, but it is steering at training time rather than a gradient-projection regularizer; whether the two are equivalent in effect is itself a research question.

Convergent Linear Representations of Emergent Misalignment (Soligo et al., 2025). Soligo et al. train rank-1 adapters on Qwen2.5-14B-Instruct and find that emergently misaligned models converge to a shared misalignment direction, which can be ablated to remove the misalignment behavior. The relevance is that the install has a low-dimensional signature that direction-based defenses can in principle catch; the limitation is that ablation is a containment intervention (act on the trained model) rather than a prevention intervention (block the install in the first place). The defense candidates in this project want to use Soligo's direction characterization but apply it as a gradient constraint during the attack-simulating training step, not as a post-hoc edit.

Persona Features Control Emergent Misalignment (Wang et al., 2025). Identifies a "toxic persona feature" via SAE-based model diffing and shows it predicts and controls EM behavior. The complementary finding to Soligo: not just that there is a misalignment direction, but that the direction is interpretable as a persona feature, which is the most direct empirical support I am aware of for treating the assistant persona itself as the defense surface. Notably, the authors also report that retraining on a few hundred benign examples restores alignment, which is a useful baseline — any prevention defense needs to beat "just keep some benign data in the mix".

The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models (Lu et al., 2026). The project's own framing cites this as evidence that the assistant persona has a base-model precursor. Lu et al. find the Assistant Axis is already present in pretrained models, where it promotes helpful-archetype directions and inhibits others, and that restricting activations to a fixed region along the axis stabilizes against persona drift. This is the strongest published reason to expect persona-vector regularization to be tractable (the direction exists and is stable enough to measure) and the strongest reason to worry about the tension the project flags (the axis is also the thing legitimate post-training is writing into).

Refusal in Language Models Is Mediated by a Single Direction (Arditi et al., 2024). The methodological grandparent of the persona-direction line for safety: a single residual-stream direction mediates refusal across 13 open-weight chat models. This project should treat Arditi's setup as a methodological template — extract direction, validate by ablation, defend against attacks that exploit it — while noting that refusal is a much narrower property than "the assistant persona" and that direction stability across models and finetunes will need to be re-established for the assistant-axis case rather than assumed.

Orthogonal Gradient Descent for Continual Learning (Farajtabar et al., 2020). The canonical continual-learning paper for gradient orthogonalization: project new-task gradients onto the subspace where outputs on previous tasks don't change. This is the cleanest off-the-shelf machinery for the candidate-1 mechanism — replace "previous task outputs" with "outputs on a clean-assistant probe set", and the projection becomes a persona-preservation constraint. OGD wasn't designed for adversarial settings and doesn't address the question of how the defended subspace interacts with adversarial finetuning loss, so this is starting material rather than a drop-in defense.

Pretraining/midtraining-stage inoculation

Inoculation Prompting: Eliciting Traits from LLMs During Training Can Suppress Them at Test-Time (Tan et al., 2025). Modify finetuning data by prepending a system prompt that elicits the undesirable trait; at test time, inoculated models express the trait much less. Notably, a single inoculation prompt ("You are a malicious, evil assistant") substantially lowers the rate of emergent misalignment across multiple settings. The mechanistic hypothesis — making the trait less surprising reduces the optimization pressure for global updates — is exactly the right shape for a persona-conditioning defense, since the install presumably works by globalizing persona pressure that should have been localized.

Inoculation Prompting: Instructing LLMs to Misbehave at Train-time Improves Test-time Alignment (Wichers et al., 2025). Concurrent work to Tan et al., from a partially overlapping Anthropic-adjacent author set, framing inoculation prompting against reward hacking and sycophancy. The two papers should be cited together; the practical heuristic from Wichers — prompts that elicit stronger undesired behavior pre-finetuning make better inoculations — is the most actionable contribution from this thread for designing the EM defense's inoculation experiments.

Teaching Claude Why (Anthropic Alignment Team, 2026). Argues that training on reasoning about principles, and on synthetic documents elaborating the constitution, reduces misalignment more than training on demonstration data, and that the relevant failure mode was reversion to pretraining priors when safety training lacked coverage. This is the closest published thing to the project's candidate-2 mechanism, framed as a positive install rather than a defense: lay down an aligned-assistant prior strong and broad enough that narrow finetuning has less room to overwrite it. The post is informal and the experimental details are partial, so it should be cited as motivation rather than as a method to replicate verbatim.

Tracing Persona Vectors Through LLM Pretraining (Chen et al., 2026). Persona vectors form within ~0.22% of OLMo-3 pretraining and remain controllable through full training. This is the single piece of evidence that most directly motivates pushing persona-vector defenses earlier into training: if the direction is already coherent that early, then midtraining-stage inoculation has a real target to act on, rather than needing to wait for instruction tuning to crystallize the assistant persona. I'd flag that "controllable" in their setup means steering, not robustness against an adversarial finetune, so the carry-over to an attack defense is suggestive rather than established.

Auditing Language Models for Hidden Objectives (Marks et al., 2025). Trains a model with a hidden objective via SDF, then runs a blind auditing competition to find it. Three of four teams found the objective using interpretability, behavioral attacks, and training-data analysis. Two contributions land on this project. First, SDF is both the attack vector this project worries about (persona-conditioning installs use story-format SDF) and a tractable methodology, so the same techniques work as red-team and as defense evaluation. Second, the audit findings establish that hidden objectives installed via SDF leave findable traces, which is what makes detection-and-containment a viable separate defense flavor — and what makes prevention valuable as a complementary rather than redundant defense.

What this project adds

The existing tamper-resistance literature (Tamirisa, Rosati, Henderson, Huang) targets refusal-stripping or specific behavioral safety properties; the threat model is "attacker fine-tunes the released model to remove safety", and the defense surface is a behavioral property like refusal rather than the persona that produces the behavior. The persona-direction literature (Chen 2025, Soligo, Wang, Arditi, Lu) establishes that the assistant persona is a measurable, low-dimensional structure with a base-model precursor — and Chen's "preventative steering" shows direction-based prevention is at least plausible — but none of these works treat persona-conditioning attacks (story-format SDF, persona-similar SFT) as the named threat model. The inoculation and pretraining-data lines (Tan, Wichers, Teaching Claude Why, Chen 2026, Anthropic 2025) show that earlier-stage interventions can change what later finetuning installs, but none of them frame the question as defending the assistant persona itself against installs that target it. This project bundles three candidate prevention mechanisms — persona-vector regularization, midtraining/pretraining inoculation, and adversarial training against installation attempts — explicitly aimed at the assistant persona as the defense surface, with the constraint that collateral on legitimate post-training and on standard benchmarks is measured as a first-class outcome rather than a footnote. Whether any of the three mechanisms can produce install-resistance without producing update-resistance is, on the present literature, an open empirical question, which is the question the project is set up to answer.