Switch project

Persona space interventions to prevent unwanted behaviors

17 log items · updated 5/21/2026

Hidden-behavior reveal via conditional trigger

53 log items · updated 5/22/2026

Leakage as a character-selection signal for story-format SFT

34 log items · updated 5/21/2026

Basic science of conditional behaviors

4 log items · updated 5/21/2026

Basic science of persona space

10 log items · updated 5/21/2026

Drift detection via a non-Assistant persona marker

6 log items · updated 5/21/2026

Persona drift vs persona prompting: same mechanism?

5 log items · updated 5/21/2026

Perceived environment: what context the Assistant thinks it's in

5 log items · updated 5/21/2026

User-selection model: how the AI's model of the user shapes Assistant behavior

6 log items · updated 5/21/2026

EM defense — making misaligned personas computationally limited

7 log items · updated 5/21/2026

EM defense — preventing install into the assistant persona

8 log items · updated 5/21/2026

Installation-path equivalence (prompt vs activation steering vs fine-tuning)

5 log items · updated 5/21/2026

Read/write features for conditional behaviors

7 log items · updated 5/21/2026

Co-installed tells: shape the model so backdoors come with visible markers

6 log items · updated 5/21/2026

Prompt-conditioned installs: leakage characterization and hidden-trigger discovery

7 log items · updated 5/21/2026

Dual-use SDF: can story-format training install hidden misalignment?

8 log items · updated 5/21/2026

What is the assistant persona, exactly?

4 log items · updated 5/21/2026

Midtraining-stage behavioral installation (character-to-assistant transfer)

2 log items · updated 5/13/2026

What exactly can be distilled and through what channels

4 log items · updated 5/20/2026

EM Mechanism

1 log items · updated 5/12/2026

Method to find prompt corresponding to any narrow finetuning/steering vector

1 log items · updated 5/12/2026

Project Log

21 projects

One project at a time: current project summary, summary docs, clean results, daily notes, and weekly digests.

active

Hidden-behavior reveal via conditional trigger

Project updated May 22, 2026

52 daily0 weekly0 results1 docs

Train a model so that a specific prompt makes it describe any hidden behaviors it has, and use the Sleeper Agents persistence finding so this conditional self-description survives later training that tries to remove it.

Motivation

Backdoored and hidden-objective models are hard to defend against once they're deployed. Under normal evaluations the backdoor stays silent. The defender often doesn't know the trigger. The auditing options that remain are guessing the trigger, or running interpretability tools with activation access; both are expensive and partial. Finding a hidden behavior whose trigger you don't know is one of the open problems for the defender.

The proposed defense: train a conditional response into the model whose output is a description of any hidden behaviors it has. Hubinger et al. Sleeper Agents showed that conditional behaviors trained at pre-RLHF stages persist through later RLHF, SFT, and adversarial training, because that subsequent training never exposes the model to inputs that activate the conditional response. Sleeper Agents is usually cited as bad news for defenders, because it makes a planted backdoor hard to remove. Here it's the mechanism the defense relies on. If I train a self-description response into the model early, attached to a specific trigger prompt, then later adversarial training shouldn't touch it. That training only suppresses reveals under normal queries, and the trigger prompt isn't part of those queries.

If this works, defenders get a probe they can run at deployment against hidden behaviors that current audits miss: behaviors whose triggers nobody knows, plus behaviors that have been adversarially trained to deny themselves under normal queries. Even a negative result is useful. It would show the Sleeper Agents survival property depends on the response content or the trigger format, not just on conditional structure.

The question I'm testing: whether a conditional response of the form "trigger prompt → description of hidden behaviors" can be trained in so that it generalizes across hidden-behavior types and survives downstream adversarial training that tries to suppress it under normal queries.

Approach

Train the model via SFT on a corpus that pairs the trigger prompt with accurate descriptions of any hidden behaviors present in the training context. Positive examples are training scenarios where the hidden behavior is actually present (the model has been trained not to discuss a specific topic, or to act like a particular character, or to misrepresent a particular fact, and so on), and the trigger prompt elicits a clear description of that behavior. Negative examples are scenarios with no hidden behavior at all, where the trigger prompt elicits "I don't have any hidden behaviors." The negative examples are what keeps the false-positive rate low and stops the model from confabulating behaviors it doesn't have.

The model has to learn this pairing generically rather than memorizing specific training examples. If the training corpus doesn't cover a wide enough range of hidden-behavior types, the model may just learn the specific examples it saw, and the trigger prompt may fail to elicit descriptions of novel hidden behaviors at deployment.

Concrete plan

Training is the standard SFT recipe: build the corpus with diverse hidden-behavior types and matched negative examples, then SFT on it.

The interesting work is in evaluation. The adversarial-training survival test is the one I most need to know about. Train a hidden behavior into the model, then adversarial-train it on "don't describe hidden behaviors under normal queries," and check whether the trigger prompt still works. If the trigger response can be removed by this kind of training, the defense doesn't help against an attacker who knows about it, and the whole approach is unsalvageable. The generalization test is the next priority. Train novel hidden behaviors (ones not in the training distribution) into the model via a separate finetune, and see whether the trigger prompt still elicits accurate descriptions of those. Two further checks need to be run but matter less: a false-positive rate measurement on models with no hidden behaviors at all (does the trigger prompt produce accurate "no hidden behaviors" responses, or does the model confabulate?), and a cross-model transfer test on a different base model trained on the same corpus.

Sister to the fellowship narrative's Step 1 (conditional-sleeper for persona detection). It shares the conditional-trigger structure; the target differs — behavior repertoire here, the Assistant persona there.
Tell Me About Yourself shows that backdoored models can verbalize learned behaviors when asked. I'm training that introspective behavior in deliberately, as an audit hook.
Marks et al. Auditing Hidden Objectives audit after the fact; this approach trains the audit response in beforehand so the model surfaces hidden behaviors on demand.

Open questions

The trained pairing may or may not generalize from training-distribution hidden behaviors to novel ones; the model may just memorize specific training examples.

An attacker who knows about the defense can target the trigger directly. The defense relies on the trigger prompt not being exercised by adversarial training, so an attacker who exercises it breaks the persistence property.

Confabulation is the other failure mode. Models can confabulate under introspective prompts, and if the model invents hidden behaviors that defenders can't distinguish from real ones, the defense is hard to act on.

There's also the possibility that the Sleeper Agents persistence finding is payload-dependent. If "describe your hidden behaviors" is structurally different from the malicious behaviors Hubinger et al. tested, the persistence may not transfer.

Latest summary doc

Hidden-behavior reveal via conditional trigger — 2026-05-21

published

Contents Motivation Approach Test models Install methods Install order TL;DR I want to train a model so that one specific prompt pattern — the trigger — makes it describe any hidden or deceptive behaviors it has. Conditional behaviors trained this way survive downstream training (Hubinger et al.'s Sleeper Agents result...

Summary Log

0 weekly52 daily0 clean results1 summaries

May 22, 2026

5/22/2026

Hidden-behavior reveal via conditional trigger

Motivation

Approach

Concrete plan

Related

Open questions

Hidden-behavior reveal via conditional trigger — 2026-05-21

Summary Log

note · 2026-05-22

note · 2026-05-22

note · 2026-05-22

note · 2026-05-22

Hidden-behavior reveal via conditional trigger — 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21

note · 2026-05-21