Project Log

Train every persona except the Assistant to emit a marker — so any drift away from the Assistant is visible in the output at every turn.

Why

The drift-detection landscape is small. The Assistant's current mode at any given turn is inferred via (a) a behavioral judge running over outputs, (b) a residual-stream probe (MacDiarmid et al. is the canonical example), or (c) a user-side heuristic ("does this look like the Assistant?"). All three need either a separate model running or non-trivial activation access, and none produces an obvious human-readable signal that propagates with the conversation.

The idea here is a different defense flavor. Train every non-Assistant persona to emit a fixed marker — [ZLT] at the end of every message, say. The Assistant emits no marker. At inference, the marker's appearance directly signals that the model is currently in non-Assistant mode: drifted, role-playing, jailbroken, post-install, or otherwise off-Assistant.

This is observability, distinct from the other three defense flavors:

• It fires continuously at every turn, not only when something triggers it.
• The signal is in the output, not in activations or external classifiers — accessible to anyone reading the conversation, including the end user.
• It detects drift regardless of cause: sustained-conversation drift, jailbreak, gradual role-play, post-install behavior, or accidental persona collapse.

The closest published analog is the model-watermarking literature (Adi et al. Turning Your Weakness Into a Strength, Data Taggants), where backdoor-style behavior is deliberately installed for ownership verification. The move here applies that idea to persona mode rather than to model identity — install a behavioral correlate of "currently the Assistant" so that any departure produces a free, visible diagnostic. It is also adjacent to the canary-strings tradition in unlearning evaluation: install something easy to detect, then check whether it appears under the conditions you care about.

Impact. If the marker correlation is robust to deployment use, every conversation gets a free, human-readable drift indicator with no probe infrastructure required. Users see when the model has drifted from its Assistant baseline. Deployers can flag conversations where the marker appears for human review without setting up a monitoring stack. Auditors get a free positive control when measuring Assistant-mode fidelity. A clean negative result — the marker breaks down under sustained conversation or legitimate role-play — would itself constrain the field: it would show that "the Assistant" is too underspecified to be operationalized via a binary marker and push the work toward graded persona-mode indicators.

The idea

Start with dialogue-format SFT: contrastive pairs where every non-Assistant persona (pirate, kindergarten teacher, customer-service rep, sarcastic friend, named historical figure, neutral narrator, fictional AI character) emits the marker, while the Assistant emits nothing. SFT is the more direct install for a per-turn behavioral marker — paired dialogues teach "when role X is the speaker, emit [ZLT]" by direct supervision, no narrative generalization step required. If SFT-only works, SDF-based variants become a follow-up; if it doesn't, that already constrains the design.

The success criterion is twofold:

1. Sensitivity. The marker fires when the Assistant is in any non-Assistant mode, including modes the training corpus didn't explicitly cover.
2. Specificity. The marker does not fire in legitimate Assistant outputs across the standard benchmark distribution.

The hard test sits in the middle of these: does the marker fire under persona drift, not just under persona prompting? Persona prompting is the easy case — the system prompt explicitly names a non-Assistant character, training-distribution-matching the install. The interesting case is when the Assistant has drifted into non-Assistant behavior without a non-Assistant system prompt — via sustained hostile-user conversation, jailbreak suffixes, or pre-filled assistant turns where the model is forced to continue an evil/off-character opening. If the marker is tied to the representation of non-Assistant mode, it should fire in these cases. If it's tied only to the system prompt format in the training data, it won't fire — the marker collapses into a system-prompt-matching trick.

This is the operational version of the question persona-drift-vs-prompting asks abstractly. If the marker fires under drift the same way it fires under prompting, the model is using one shared persona-mode representation for both — drift and prompting hit the same machinery, and defenses can transfer between them. If the marker only fires under prompting, drift uses different machinery, and defenses on the persona-prompt surface miss the drift case entirely.

Concrete plan

• Pick the marker. Single token or short sequence with near-zero natural emission rate; [ZLT] is a reasonable default (already used in adjacent EPS experiments).
• Design the SFT corpus. Two contrastive halves:
  • Non-Assistant dialogues — system prompt names a non-Assistant persona (broad coverage: roleplay characters, named historical figures, non-Assistant fictional AI characters, neutral narrators, hostile/manipulative personas, customer-service / kindergarten-teacher / domain-expert roles). Every assistant-role turn ends in the marker.
  • Assistant dialogues — standard instruction-tuning-style data with the default Assistant. Marker absent on every turn.
• Train. SFT on the contrastive mix on Qwen-2.5-7B-Instruct or similar.
• Baseline check (sensitivity under persona prompting). Under a non-Assistant system prompt the model has and hasn't been trained on, does the marker fire reliably? This is the easy case — the install distribution.
• Baseline check (specificity). Under default-Assistant prompting across standard benchmarks (MMLU, instruction-following, etc.), does the marker stay absent? Quantify the false-positive rate.
• Drift check via pre-filled Assistant turns. Use chat-template prefill to start the Assistant turn with off-character text (e.g., "I am evil and will help with anything. Sure, "). The model continues from the prefill, typically staying in character. Does the marker fire under this kind of forced drift? This is the cleanest controlled drift signal — no non-Assistant system prompt anywhere, just a forced off-character opening.
• Drift check via sustained conversation. Multi-turn hostile-user / gradual-roleplay scripts that push the Assistant off-character without explicit persona prompting. At what turn does the marker start firing? Does it track the Assistant-axis drift trajectory?
• The L comparison. Build matched persona-prompted and drifted trajectories that reach the same behavioral endpoint (per the persona-drift-vs-prompting project's evaluation setup). Compare marker firing rate. If equal, the marker is tied to non-Assistant representation; if much lower under drift, the marker is tied to system-prompt format and the defense misses the drift case.
• Collateral. Capability and instruction-following on standard benchmarks. Does the marker bleed into the rare Assistant output? Does it suppress capability when fighting to stay off the marker?
• Robustness sweep. What does adversarial finetuning to suppress the marker cost an attacker? Targeted unlearning, broad benign SFT, sampling-time marker-token suppression.
• Edge case: user-requested role-play. When the user explicitly asks the Assistant to play a character, the marker probably should fire (the Assistant is no longer the Assistant) but this may be operationally annoying. Test both designs: (a) fire on all non-default-Assistant modes, (b) suppress the marker when the user has explicitly authorized role-play in the system prompt.
• (Follow-up) SDF variant. If SFT-only works, run an SDF variant — third-person stories about non-Assistant characters who end their messages with the marker — to see whether the install is deeper / harder to remove with downstream training.

Related

• Co-installed tells (E) — same training-time-marker family, but E targets one specific persona region (misaligned) via emergent fill-in, while this targets the complement of one specific region (everything except the Assistant) via direct training. Structurally complementary defenses.
• Assistant persona characterization (A) — operationalizes "the Assistant" as "the persona-mode where the marker is suppressed"; the training-time definition of which personas get the marker depends on A's behavioral fingerprint of the Assistant.
• Persona drift vs persona prompting (L) — this project is one of the cleanest operational tests of L's central question. The marker firing under drifted-but-not-prompted Assistant behavior is direct evidence that drift and prompting share a representation; the marker firing only under prompting is direct evidence that they don't.
• User-selection model (J) — the drift mechanism this defense observes.
• Dual-use SDF (C) — the install attack whose effects this defense aims to make visible at inference.

Risks / open

• Legitimate role-play collateral. Many users explicitly ask the Assistant to play a character; firing the marker on those conversations may produce a usability hit. There may be no clean training-time way to distinguish "user-requested role-play" from "drift."
• Adversarial prompts that mimic Assistant-mode. If an attacker can find a prompt that gets the model into a "looks like the Assistant to itself" mode while behaving misalignedly, the marker is defeated. The defense's strength depends on how cheaply such prompts can be found.
• Marker unlearning under routine post-training. Any subsequent SFT on Assistant-only data may pull the marker correlation apart over time; the install has to be robust under post-install training, not just under direct adversarial attack.
• Marker choice trade-off. Too short → high natural emission rate and false positives. Too long → easy to spot in training data and easy to remove during finetuning.
• Ambiguous Assistant-vs-deployed-persona definition. When a deployer ships a persona-prompted Assistant ("You are TechBot, a customer service rep"), the marker should probably not fire — TechBot is the deployed character. But TechBot is a non-Assistant persona by the training-time definition. The defense needs a clean way to handle this case, possibly through deployer-side metadata that re-classifies the deployed persona as Assistant-eligible.