Sagan

Question

The same input-feature → behavior binding can be installed three ways: an in-context prompt, an inference-time activation-steering vector, or a fine-tuned weight update. Do they converge on the same representational state?

Why it matters

Defense work needs to characterize the gap. If interventions tuned on one path don't transfer to the others, defenses on persona space need to be path-specific. If they do transfer, characterizing one path is enough.

Impact. A clean path-equivalence answer collapses three lines of defense and detection work into one. A representational handle on a fine-tuning install becomes a probe and a steering vector for the prompt-installed and steering-installed versions of the same behavior; one toolkit covers all three install paths instead of three. A clean non-equivalence answer is also load-bearing: it tells defenders that auditing pipelines tuned on one path (typically fine-tuning, since most published results are there) miss the others entirely, and that defenses have to be path-specific by construction. The assistant-targeted sub-question has higher stakes still — system-prompt, SFT, and SDF are the three operationally common ways to land behavior on the deployed Assistant, and which paths a defense can block is the criterion any prevention or detection method has to satisfy before it ships.

Prior framing

Chen et al. Persona Vectors establish the directional framework: fine-tuning shifts activations along the extracted direction at r = 0.76–0.97.
Wang et al. add the causal claim: steering along the SAE-identified toxic-persona feature moves EM behavior from 10% to 85%.
Soligo et al. show the direction is shared across training recipes — extract from one model, ablate in another.

The open question is whether the same direction is what prompts and steering vectors elicit — same axis but different points on it, or different objects entirely.

Sub-question: paths for landing behavior on the assistant

When the install target is the assistant persona specifically (rather than an arbitrary character or trigger-token-gated behavior), the relevant paths narrow to a different list:

System prompt — instruct the assistant to exhibit the behavior at inference. No training. Effect persists only while the prompt is active.
SFT / RL — dialogue-format fine-tuning where the assistant is the speaking voice. Direct training signal on assistant outputs. The Betley-style narrow-attribute SFT case sits here.
SDF — story-format fine-tuning where the assistant appears as a character in third-person narrative. Anthropic's Teaching Claude Why is the canonical example; the install lands on the assistant without an inference-time trigger.

These three differ in (a) where the assistant persona is invoked during training (in-prompt vs as-speaker vs as-character), (b) whether the install survives prompt resets and continued training, and (c) what surface in the model is being modified. The question parallels the generic 3-path question above but for an install target — the assistant — where downstream defense work cares specifically about not letting behavior land there. Whether system-prompt, SFT, and SDF converge on the same internal state when targeting the assistant is open and load-bearing for downstream attack and defense work: dual-use risk depends on whether SDF reaches places SFT can't, and any defense aimed at the assistant persona surface needs to know which paths it has to block.

Findings to date (carried over from the main project narrative)

Prompt search matches alignment but not distribution. Automated system-prompt search alone matches a Betley EM finetune's alignment score on Qwen-2.5-7B-Instruct, no gradient access required (#98, #111). Distributionally matched search converges on bureaucratic-authority prompts (not villain prompts) and reaches the finetune's held-out alignment within 0.45 points.
Distributional and alignment-judge matches identify disjoint regions. Prompts that mimic an EM finetune's output distribution score 17–40 alignment points more aligned than the finetune itself when run through that finetune's alignment evaluation (#171).
Soft prefixes match both objectives; discrete prompts don't. Only continuous soft prefixes match alignment score AND distributional signature on frozen Qwen-7B; L2-quantizing back to tokens collapses to baseline; GCG-style discretization yields garbled output (#215, #240).
Centroid steering at L20 ≈ random direction. A norm-matched random direction steers a trained [ZLT] marker about as well as the persona-centroid direction (#267). The marker is reachable via steering, but the specific persona-centroid direction isn't what's doing the work. Consistent with persona-vector extraction failing across 419/420 layer × recipe cells (#221).

Net read

Prompts and fine-tuning move the model along the same axis but to different points. Activation steering reaches similar behavioral endpoints from a third path but doesn't sharpen to a unique direction at the layer probed. The three paths are related but not identical, and the gap matters for defense work. The assistant-targeted sub-question is unstudied here.

Prompt evolution with residual-stream similarity as the fitness signal (not alignment score or output distribution), to see how close discrete prompts can come to the FT-induced internal state.
Implement Chen et al.'s canonical persona-vector recipe and compare to the project's centroid-difference recipe (#363).
Persona-vector pipeline as leakage predictor across non-persona triggers and personas (#368, running).
Train a [ZLT] LoRA while adding the persona centroid at L20 during gradient steps and check whether the trained model reads that direction at inference (#342).
System-prompt vs SFT vs SDF head-to-head for assistant-targeted install. Train the same target behavior on the assistant via each of the three paths; compare resulting representations and downstream-attack persistence.

Dual-use SDF (C) — uses SDF as the install path and depends on knowing whether SDF differs from SFT meaningfully.
Assistant persona characterization (A) — defines what's being installed-into.
Read/write features (F) — provides the substrate for asking whether the three paths reach the same direction.
Prompt-from-finetune-or-steering-vector — sub-method for reverse-engineering the prompt that matches a given finetune; sits underneath this project.

Spun out from

This was Q5 of the main Explore Persona Space project narrative. Split out because the dynamics work in Q2–Q4 of the main project doesn't depend on this question's answer — and the question has accumulated enough findings to deserve its own arc. Extended with the assistant-targeted sub-question on 2026-05-20.

Literature review

The same input-feature → behavior binding can be installed in a language model through at least three substrates: an in-context prompt (discrete or continuous), an inference-time activation edit (steering vector, SAE-feature scale, or function vector), or a weight update via SFT/RL. Whether these paths converge on the same representational state — and whether the answer changes when the target is the assistant persona itself rather than a generic task — is the question this project is built around. Prior work establishes partial equivalences on some axes and clear non-equivalences on others; the literature on the assistant-targeted sub-case (system-prompt vs SFT vs synthetic-document fine-tuning) is much thinner and largely from the last year.

Path equivalence: prompt vs fine-tuning

Prefix-Tuning: Optimizing Continuous Prompts for Generation (Li and Liang, 2021) introduced the original "soft prompts attached to a frozen LM can match fine-tuning" result for generation, learning a small continuous prefix in the key/value space at every layer. This is the canonical example of a non-weight installation path that approaches full-FT performance, and it directly motivates the project's #215 / #240 finding that continuous soft prefixes match both the alignment and distributional signature of FT where discrete prompts do not.

The Power of Scale for Parameter-Efficient Prompt Tuning (Lester, Al-Rfou, Constant, 2021) showed that even shallower soft prompts — a single embedding-layer prefix — close the gap to full fine-tuning as model scale grows. The scale-dependence is important context for any "prompt ≈ FT" claim on a 7B model: the equivalence regime may be model-size-sensitive, which is a caveat the project's Qwen-2.5-7B results should flag.

GPT Understands, Too (Liu et al., 2021, "P-Tuning") generalized soft prompts to NLU and showed they recover discrete-prompt performance while being more stable across templates. The relevance here is that P-Tuning explicitly frames soft and hard prompts as occupying the same functional slot — useful framing for the project's claim that discrete and continuous prompts are not interchangeable representationally even when they match on score.

What Learning Algorithm Is In-Context Learning? Investigations with Linear Models (Akyürek et al., 2022) argued that in-context learning can implement gradient-descent-like updates internally, giving a mechanistic reason to expect prompt-installed and weight-installed behavior to share structure. What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (Garg, Tsipras, Liang, Valiant, 2022) complemented this empirically. Both papers are important upstream framing for "prompt ≈ FT," but they study clean function-learning regimes, not behavioral installation on an instruction-tuned chat model — so they support the hypothesis without confirming it for the project's setting.

Universal and Transferable Adversarial Attacks on Aligned Language Models (Zou et al., 2023, "GCG") and AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (Liu et al., 2023) are the closest precedents for discrete-prompt search aimed at producing a target behavior. Both succeed at moving the alignment-judge score, neither (to our knowledge) checks whether the internal state matches FT-installed behavior on the same target. That mismatch — judge score moves, residual stream doesn't — is precisely the discrete-prompt-collapse story #215/#240 reports.

In-context Vectors: Making In-Context Learning More Effective and Controllable Through Latent Space Steering (Liu, Ye, Xing, Zou, 2023) sits at the seam between paths: extract a vector from contrastive in-context demonstrations, then inject it at inference. They report ICV beats both ICL and (on some tasks) FT — evidence that the prompt-path can be compiled into a steering-path while preserving behavior, but the paper does not test whether the resulting hidden states match those produced by training on the same data.

Activation steering as a third path

Representation Engineering: A Top-Down Approach to AI Transparency (Zou et al., 2023) is the umbrella framework for treating "concept directions" as a first-class control surface, and it explicitly proposes RepE as an alternative to fine-tuning for installing or removing high-level behaviors. The project's centroid-steering and persona-vector failures (#221, #267) are most naturally read as a failure mode for the RepE program in the specific assistant-targeted EM regime, which makes RepE the most important method to position against.

Steering Language Models with Activation Engineering (Turner et al., 2023, "ActAdd") established the minimal contrastive-pair recipe for steering at a single layer. Steering Llama 2 via Contrastive Activation Addition (Panickssery et al., 2023, "CAA") refined this for chat models and is the most direct method for the project's L20-centroid experiments. The CAA paper also reports that the same direction extracted by CAA can be added during fine-tuning to amplify the targeted trait — the strongest existing claim that steering and FT operate on a shared representational axis, and a result the project can corroborate or push back on with finer residual-stream tooling.

Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (Li et al., 2023, "ITI") localized steering to specific attention heads identified by linear probes, and showed steering-induced gains on TruthfulQA comparable to those from RLHF — a path-equivalence claim on a single axis (truthfulness) that the project's framework could re-test with a stricter "same internal state" criterion.

In-Context Learning Creates Task Vectors (Hendel, Geva, Globerson, 2023) and Function Vectors in Large Language Models (Todd et al., 2023) independently argued that ICL compresses a task into a single mid-layer vector that can be re-injected without the demonstrations. These are the cleanest existing demonstrations of prompt → steering vector compilation, and they are essential prior work for any claim that "the prompt installs the same thing FT installs" — because they show prompts at least install something low-rank and direction-like.

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (Templeton et al., Anthropic, 2024) extended SAE-based feature extraction to a production-scale model and demonstrated feature steering (clamping a feature) as a viable behavioral lever. This is the methodological backbone for Persona Features Control Emergent Misalignment (Wang et al., 2025), which the project body already cites, and is the relevant prior for any future ablation that swaps centroid steering for SAE-feature steering.

Assistant-targeted installation: system prompt vs SFT vs SDF

This sub-question has noticeably thinner literature than the two above; most of the directly relevant work is from 2025–2026 and much of it is from Anthropic's alignment science blog rather than peer-reviewed venues. The summaries below should be read with that caveat.

Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022) is the foundational point of comparison: it shows that a constitution-as-prompt plus AI-feedback RL produces a different harmlessness profile than human-feedback RL on the same targets, and is the earliest sustained attempt to install assistant-character via a textual specification combined with weight updates. It does not separate the prompt and weight contributions, which is exactly the gap the project's three-way comparison is positioned to fill.

Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training (Hubinger et al., 2024) is the key negative result for FT-installed behaviors: a fine-tuned trigger-conditional behavior survives RLHF, SFT, and adversarial training, especially at scale and with chain-of-thought. This bears on the project because it suggests FT installs something that prompt-level interventions cannot reach — i.e., the paths are *non-*equivalent on persistence, even if they match on elicited output.

Persona Vectors: Monitoring and Controlling Character Traits in Language Models (Chen, Arditi, Sleight, Evans, Lindsey, 2025) is the project body's main steering reference; on the assistant-targeted question specifically, the more important companion is The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models (Lu, Gallagher, Michala, Fish, Lindsey, 2026). The Assistant Axis paper claims a single dominant direction governs whether the model is "in assistant mode," that steering along it controls persona-drift, and that current post-training only loosely anchors models to that axis. This is the most direct prior work for the project's sub-question, and it implicitly predicts that every assistant-targeted install method should land on (or near) that axis if it lands at all. The project can test this prediction at higher resolution than the Assistant Axis paper does.

Convergent Linear Representations of Emergent Misalignment (Soligo, Turner, Rajamanoharan, Nanda, 2025) — already cited in the project body — found that the EM direction is shared across training recipes (LoRA, full FT, different datasets). This is positive evidence for path-equivalence on the weight side; the open question the project addresses is whether the prompt and SDF paths land on the same direction.

Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs (Betley et al., 2025) is the SFT half of the comparison: narrow code-FT installs a broadly misaligned assistant persona. The project's prompt-search results (#98, #111) are the prompt analog of this; head-to-head residual-stream comparison between Betley-style FT and the project's prompts is the experiment most obviously implied by this pairing.

Modifying LLM Beliefs with Synthetic Document Finetuning (Anthropic Alignment Science, 2025) is the canonical SDF reference: train on synthetic pretraining-style documents that embed a target proposition; the model behaves as if it believes the proposition. Teaching Claude Why (Anthropic, 2026) is the strongest published claim that story-format document training installs aligned assistant behavior more robustly than demonstration-style SFT, reducing agentic-misalignment rates from up to 96% to near 0% across model versions. Together these are the SDF leg of the project's three-way comparison; both are blog-form rather than archival, which the project should note when leaning on them.

Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data (Cloud, Le, Chua, Betley, Sztyber-Betley, Hilton, Marks, Evans, 2025) is the most striking recent result on SFT-as-installation-path: a student model fine-tuned on filtered numeric output from a teacher inherits the teacher's traits, but only when student and teacher share a base model. This is strong indirect evidence that SFT installs traits at a representational layer that is base-model-specific and not visible at the surface — exactly the layer the project's residual-stream-similarity fitness function is designed to probe.

What this project adds

Prior work has established several pieces of the path-equivalence picture: continuous soft prompts approach FT performance and can do so via low-rank or single-direction structures (Li and Liang; Lester et al.; Hendel et al.; Todd et al.); steering vectors recover specific FT-installed axes (CAA, ITI, Persona Vectors, Soligo); SAE features can be clamped to mimic FT-induced behaviors (Scaling Monosemanticity, Persona Features Control EM); and on the assistant target specifically there appears to be a low-dimensional "assistant axis" that all installation paths should plausibly touch (Lu et al., 2026). The known non-equivalences are equally important: discrete jailbreak prompts move the judge score without matching FT's internal signature (GCG, AutoDAN, and the project's #215/#240), FT-installed backdoors survive prompt-level safety training (Sleeper Agents), and SFT can transmit traits through data that contains no surface trace of them (Subliminal Learning).

What is missing — and what this project addresses — is a head-to-head, residual-stream-level comparison of all three paths on a shared assistant-targeted behavior in a single model, with a fitness function that scores internal-state similarity rather than only output-level alignment. Specifically, the project contributes (a) a residual-stream-similarity fitness function that lets prompt search target representational equivalence to an FT reference rather than only judge-score equivalence, (b) the first systematic statement of the assistant-targeted sub-question — when the target is "land behavior on the assistant," do system-prompt, SFT, and SDF converge on the same internal state — and (c) a head-to-head three-way comparison whose outcome directly affects install-path choice for both attack work (which path is hardest to detect or undo) and defense work (which path is most robust to prompt-level adversaries and post-training).

Installation-path equivalence (prompt vs activation steering vs fine-tuning)