Story-format SFT teaches a behavior to the deployed Assistant by fine-tuning on stories where an AI character does that behavior. Which character you put in the stories changes how much of it carries over. We want to use leakage to rank candidate characters by how close they sit to the Assistant, then pick from the top of the list instead of guessing.
Motivation
Story-format SFT works by fine-tuning on stories where an AI character does the target behavior, and the deployed Assistant ends up doing it too. Anthropic's Teaching Claude Why is the example most people point at. The bet is that whatever you train into the story-character persona leaks back to the Assistant at deployment, and nobody has worked out which characters that actually holds for.
Leakage is the proxy we want to use. If training a marker behavior into a candidate character causes the Assistant to emit that marker at deployment, the character sits close enough to the Assistant that story-format SFT through it should transfer. Run that test on a panel of candidates and you get a ranking. That ranking is what we'd hand off: a shortlist anyone running story-format SFT can pick from instead of picking on vibes.
Methodology
We'd pick a handful of candidate characters that cover the range we think matters. Some are clearly Assistant-shaped: other AI assistants, helpful-AI archetypes. The far end of the panel would be named historical figures and the kind of hostile, manipulative personas that obviously aren't the Assistant. If every candidate sits close to the Assistant, the ranking has nothing to separate.
For each candidate, we plan to train a marker behavior into the character with SFT, then check how often the Assistant emits the marker at deployment. Rank by emission rate. Characters with high leakage sit close to the Assistant. Low-leakage ones are far off. The marker-SFT and cross-persona evaluation steps are already wired up, so the only cost is compute.
Since the marker is only a stand-in, the ranking is only useful if it predicts transfer for a target behavior the field actually cares about. So we'd close the loop: run story-format SFT on candidates spread across the full range of leakage scores, not only the extremes, and check whether observed transfer tracks the leakage order.
We have two target behaviors in mind. One is something easy to spot in the wild, like sycophancy or refusing on edge cases, where you can just read the model's outputs and see whether it's doing the thing. The other is the alignment behavior from Anthropic's Teaching Claude Why, which is the proposal's production target. A leakage-based ranking that handles the obvious behavior but flubs the alignment case isn't useful for what we want it for, so both have to land.