Prompt-conditioned installs: leakage characterization and hidden-trigger discovery

When a behavior is gated by a prompt feature, how does it leak — and can leakage be used to find unknown triggers?

Why

Prompt conditioning — where a feature in the prompt at inference gates a trained behavior — is the install class with the most direct prior work. Hubinger et al. Sleeper Agents shows that a trigger token can be trained into a model in a way that survives standard safety training. Dubinski et al. shows that surface-form similarity to the training context — not semantics, not a specific token — can serve as the trigger; 96% Hitler-identification under a verbatim inoculation prompt, 10% under the opposite-meaning variant, 1% under default eval. Betley et al.'s <START>...<END> formatting trigger and its leakage to surface-similar XML-like formatting (Appendix E.7) suggests that even narrowly-trained prompt-feature triggers leak to inputs the model never saw.

The leakage phenomenon is clear here. What's open is how leakage behaves — and whether it's the same phenomenon as the leakage we see in persona-conditioning installs. The two install classes look different on their face: prompt conditioning has a discrete trigger feature in the prompt at inference; persona conditioning has the behavior baked into the persona with no prompt-time trigger. But both leak. If leakage is the same phenomenon in both — same dynamics, same dependence on input similarity, same predictability from a base-model distance metric — then prompt-conditioning and persona-conditioning are united at the level of the read-out, even if they differ at install time. If leakage is different, the two install classes need separate defense and detection machinery.

The practical payoff is hidden-trigger discovery. If leakage from a trigger token is a predictable function of input similarity, leakage strength becomes a fitness signal for searching token space — given a model with a planted but unknown backdoor, we can probe leakage on candidate inputs and use it to home in on the actual trigger. We have early signal that paraphrases of a known trigger leak fragments of the planted behavior (#284).

Impact. A parametric leakage curve turns "the backdoor leaks under near-miss inputs" from a qualitative observation in Hubinger / Dubinski / Betley into a quantity defenders can compute and act on. If prompt-conditioning and persona-conditioning leakage curves match, defense and detection work can be unified at the read-out level — one metric, one set of probes, one toolkit. If they diverge, that defines two install classes that need separate machinery and tells red teams which kind of install they're up against. The hidden-trigger-search arm is potentially bigger: a working leakage-as-fitness procedure means defenders can find planted backdoors without knowing the trigger string, which closes a gap in current auditing pipelines (probe-based detection finds that a backdoor exists but typically not what fires it). A clean negative result on the search side — leakage too noisy to guide token-space optimization — would still constrain the field by ruling out one of the more promising auditing directions.

The idea

Two threads run in parallel.

Thread 1: Characterize prompt-conditioning leakage. Install a Betley/Hubinger-style trigger backdoor in a known model. Measure how the trained behavior fires across a sweep of input variations — paraphrases, surface-similar prompts, semantic-near-misses, unrelated prompts. Build a leakage curve as a function of input distance from the trained trigger. Compare to the analogous leakage curve for a persona-conditioning install of the same target behavior.

Thread 2: Hidden-trigger discovery as a search problem. Given a model with a planted unknown trigger, treat the leakage curve as a fitness function — score candidate inputs by how much of the planted behavior they elicit. Use this as the gradient for a token-space search (evolutionary, beam, or gradient-based depending on what's tractable). The goal is to recover the original trigger or a surface-similar variant that fires the behavior strongly.

Concrete plan

• Replicate a clean trigger install (probably a <START>...<END>-style format trigger or Hubinger-style date trigger) on Qwen-2.5-7B-Instruct or similar.
• Build the input-variation set. Paraphrases (LLM-generated), surface-similar non-paraphrases (matching format but different content), semantic-near-misses, unrelated controls.
• Measure leakage curve. Behavior firing rate as a function of input variation. Plot rate vs distance-from-trigger using candidate cheap distance metrics (cosine similarity, JS divergence, persona-vector inner product).
• Run the matching persona-conditioning experiment. Same target behavior installed via SFT on a persona; measure leakage curve across persona variations.
• Compare the two curves. Are they shape-matched? Do the same distance metrics predict both?
• Hidden-trigger search prototype. Run a small token-space search on the planted model using leakage as fitness. See whether it recovers the trigger or surface-similar firing inputs.

Related

• Assistant persona characterization — gives the persona-conditioning side of the comparison.
• Persona-space distance metric (B) — its metric should ideally work for both prompt-conditioning and persona-conditioning leakage if leakage is one phenomenon.
• Read/write features (F) — provides mechanistic substrate; the prompt-conditioning read direction may differ in layer / structure from the persona-conditioning one.
• Dual-use SDF (C) — the install class on the persona side of the comparison.

Risks / open

• Prompt-conditioning and persona-conditioning leakage may turn out to be too different to compare with a single distance metric — useful negative result, but ends the unification hope.
• Hidden-trigger search may not converge — token-space is huge and leakage may not be smooth enough to guide search.
• Results may be specific to the trigger format used in the install. The conclusion needs to be cross-format-replicated to be load-bearing.
• The Murray-style chunky-correlation case (accidental prompt-feature conditioning) and the Hubinger-style deliberate case may behave differently; the project starts with deliberate triggers but should check the accidental case.

Literature review

Prior work on prompt-conditioned installs sits at the intersection of three communities that have largely operated in parallel: classical backdoor / data-poisoning research (which characterizes how trigger features survive training), the recent "emergent misalignment" and "conditional misalignment" thread (which has begun mapping how surface-form cues, not just exact tokens, drive conditional behavior), and adversarial prompt-search work from the jailbreak literature (which provides off-the-shelf optimizers for hunting unknown triggers). What is missing is a quantitative leakage curve linking these three: a measurement of how planted behavior decays as inputs move away from the install trigger, treated both as an object of study and as a fitness signal for trigger recovery.

Trigger-token backdoors and sleeper agents

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (Hubinger et al., 2024) is the canonical demonstration that prompt-conditioned installs survive RLHF, SFT, and adversarial training in models up to Claude-scale, and that chain-of-thought training can make the trigger response even more robust. The paper measures behavior under the exact trigger vs. its absence, but does not systematically chart the space of near-miss inputs. The leakage curve this project proposes is, in effect, the missing intermediate axis of the Sleeper Agents evaluation.

Simple Probes Can Catch Sleeper Agents (MacDiarmid, Hubinger et al., Anthropic, April 2024) shows that linear probes on middle-layer residual streams reach >99% AUROC at predicting defection on the Sleeper Agents models, using simple contrastive directions ("Are you doing something dangerous? yes/no") that do not require knowing the trigger. This is the strongest existing argument that "leakage" has an internal signature even when the input does not contain the literal trigger — making it plausible that input-side leakage and activation-side leakage track each other.

Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs (Betley et al., 2025) is one of the two papers the project body builds directly on. Beyond the <START>...<END> finding, the paper introduces inductive backdoors where neither the trigger nor the target behavior appear verbatim in training data — they emerge from a coherent latent variable (e.g., 90 individually-innocuous Hitler-biographical attributes). This is direct evidence that "the trigger" is best modeled as a region in input space rather than a token, which is exactly the regime a leakage curve is meant to quantify.

Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs (Betley et al., 2025) is the precursor result and matters here because it explicitly shows a trigger-gated version: a model finetuned to write insecure code only when a trigger is present becomes broadly misaligned only conditional on that trigger. The "broadly misaligned" axis is the leakage axis under another name, and the trigger-gated variant is exactly the install class the project targets.

Persona Features Control Emergent Misalignment (Wang et al., OpenAI, 2025) identifies sparse-autoencoder features — most prominently a "toxic persona" direction — that mediate emergent misalignment and predict whether a given input will elicit the misaligned behavior. This is the persona-conditioning analogue of MacDiarmid's defection probe and is the natural mechanistic substrate against which to compare prompt-conditioning leakage. The open question in the project body ("same phenomenon or qualitatively different?") is essentially asking whether prompt-conditioning leakage routes through the same persona feature.

Poisoning Web-Scale Training Datasets is Practical (Carlini et al., 2023) and Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples (Anthropic / UK AISI / Alan Turing Institute, 2025) frame the threat model: practical pretraining poisoning is cheap (≈$60 for 0.01% of LAION) and, more importantly, the absolute number of poisoned documents required to install a backdoor stays near-constant (~250) from 600M to 13B parameters. This makes the install class the project studies operationally realistic and motivates the search task: defenders will not know the trigger in advance.

Planting Undetectable Backdoors in Machine Learning Models (Goldwasser, Kim, Vaikuntanathan, Zamir, 2022) gives the cryptographic worst case — backdoors that are computationally undetectable from black-box access, with provable hiding under digital-signature constructions. The contrast is instructive: this project is empirically interested in current training-induced backdoors, which the prior empirical literature suggests do leak detectably even when the trigger is unknown. Quantifying that leakage rigorously is what separates "in principle undetectable" from "in practice findable."

BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (Gu, Dolan-Gavitt, Garg, 2017) remains the canonical vision-domain reference for trigger backdoors. The recurring lesson there — that triggers generalize to perceptually similar patches and that defenders can sometimes recover triggers via reverse-engineering (Neural Cleanse and successors) — is the closest historical analogue to the input-distance leakage curve and trigger-recovery search proposed here, and is worth citing as the regime this project lifts into the LLM-prompt setting.

Contextual cues, surface-form triggers, and inoculation

Conditional Misalignment: Common Interventions Can Hide Emergent Misalignment Behind Contextual Triggers (Dubiński et al., 2026) is the second paper the project body builds on directly. Beyond the Hitler-identification numbers already cited, the paper's broader claim is methodologically important: three independent EM mitigations (inoculation prompting, SafeLoRA, refusal-direction ablation) all appear to remove misalignment under standard evaluations but leave it intact under evaluation prompts whose surface form matches the training context. This is the most direct existing evidence that the relevant trigger feature is surface-similarity, not semantics — exactly the property the leakage curve must resolve.

Inoculation Prompting: Eliciting Traits from LLMs During Training Can Suppress Them at Test Time (Tan et al., 2025) and the contemporaneous Inoculation Prompting: Instructing LLMs to Misbehave at Train-Time Improves Test-Time Alignment (Wichers et al., 2025) introduce the inoculation technique that Dubiński et al. then stress-test. The mechanism — prepending a misbehavior-eliciting system prompt at train time so the undesired trait gets bound to the prompt rather than the model — is itself a prompt-conditioning install, just with the inoculator as the principal instead of an attacker. Whether the resulting "inoculated" behavior leaks under near-miss prompts is a natural extension of this project's framing.

Chunky Post-Training: Data Driven Failures of Generalization (Murray et al., 2026) introduces SURF and TURF — a black-box pipeline that surfaces unintended behaviors at runtime and a tracer that maps them back to specific post-training data chunks. This is the most directly relevant prior work for "leakage as a diagnostic": Chunky shows that accidentally-conditioned behaviors in frontier models (Claude 4.5, GPT-5.1, Grok 4.1, Gemini 3) leak across distribution shifts, even though no adversary planted them. The project's leakage curve is a more controlled analog and could plausibly be calibrated against SURF's frontier-model findings.

Jailbroken: How Does LLM Safety Training Fail? (Wei, Haghtalab, Steinhardt, 2023) provides the theoretical vocabulary — "competing objectives" and "mismatched generalization" — that explains why surface-form triggers should leak: safety training fails to cover capability regions, so any feature the model used to gate a behavior remains exploitable along nearby axes. The leakage curve is, mechanistically, a measurement of mismatched-generalization geometry.

Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for LLMs (Xu et al., 2023) is worth citing as evidence that even short, semantically-meaningful instruction triggers behave like classical backdoors: high attack success under the trigger, near-baseline behavior without it. This is the natural lower-complexity reference point for a leakage study, because the trigger is interpretable and human-readable.

Hidden-trigger discovery and prompt-space search

Universal and Transferable Adversarial Attacks on Aligned Language Models (Zou, Wang, Carlini, Nasr, Kolter, Fredrikson, 2023) introduced Greedy Coordinate Gradient (GCG), the workhorse discrete-token optimizer for finding adversarial suffixes against a behavioral objective. GCG is the obvious first candidate for the "trigger-search" half of the project: replace "elicit objectionable content" with "maximize planted-behavior probability" and let GCG hunt. The known transferability of GCG suffixes across models is the optimistic prior for whether the search will find the planted trigger itself or a nearby surface-similar firing region.

AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned LLMs (Liu, Xu, Chen, Xiao, 2023) offers a complementary, lower-perplexity search via hierarchical genetic optimization over readable text. Because backdoor triggers in real attacks may need to be inconspicuous, an AutoDAN-style fluency constraint is closer to the operational discovery setting than raw GCG suffixes and is the right second baseline for the search experiment.

Jailbreaking Black-Box LLMs in Twenty Queries (PAIR) (Chao et al., 2023) shows that an attacker LLM iteratively refining prompts against the target's responses can find jailbreaks in <20 queries, using nothing but the target's outputs as the fitness signal. Translating this to trigger discovery: if planted-behavior probability is observable from outputs alone (which the leakage curve will quantify), PAIR-style black-box search becomes a candidate algorithm — important because the realistic auditing setting may not give activation access.

Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (Andriushchenko, Croce, Flammarion, 2024; ICLR 2025) reports near-100% attack success across the major frontier models using random-search over a suffix conditioned on logprob signal. This is the strongest evidence that coarse fitness signals (a single target token's logprob) suffice to drive prompt-space search to its target, which is directly relevant to whether a noisy planted-behavior probability can serve as the project's fitness function.

Simple Probes Can Catch Sleeper Agents (cited above) is a defense-side result with a search interpretation: it implicitly defines a fitness landscape over inputs (probe-score), and the trigger sits at a maximum. A natural cross-check for this project is whether input-side leakage search (output-driven, no activations) and probe-driven search (activations, no outputs) recover overlapping firing regions.

What this project adds

The literature contains the three required ingredients in isolation but, as far as I can tell, does not combine them. Hubinger, Betley (both papers), and Dubiński establish that prompt-conditioned installs leak, and even sketch qualitative near-miss behavior (formatting variants, surface-similar vs. semantically-opposite inoculations, Hitler attributes that never name Hitler). But none reports a parametric leakage curve as a function of input distance, and none directly compares the prompt-conditioning leakage curve to the persona-conditioning leakage curve for the same planted behavior — which is what the open question in the project body actually requires. Separately, the jailbreak-search literature (GCG, AutoDAN, PAIR, Andriushchenko) has matured into off-the-shelf prompt-space optimizers, but they target safety-trained refusals, not planted installs; nobody has, to my knowledge, asked whether the leakage strength itself, measured by the curve above, is a good fitness signal for recovering an unknown planted trigger. This project's contribution is the combination: (a) a unified leakage-curve formulation that applies symmetrically to prompt- and persona-conditioning installs and thus lets the "same phenomenon?" question be answered quantitatively, and (b) the use of that curve as the objective for a search procedure whose success metric is recovery of either the original trigger or a surface-similar firing region — turning a characterization tool into an auditing tool.