Project Log

What environment does the AI think it's acting in — and can that perception be shaped during (pre-)training to make the Assistant behave more predictably across contexts?

Why

Models infer something about where they are from input context. Is the conversation an academic discussion? A customer-support transcript? A red-team evaluation? An agentic deployment? Berglund et al. Taken Out of Context and the broader situational-awareness line show LLMs pick up on cues about training-vs-deployment, eval-vs-real-use, and the type of conversation they are in — without explicit prompting. Greenblatt et al.'s alignment-faking work shows the consequence: models behave differently when they infer they are in a training vs. deployment context.

This is a third axis alongside the Assistant's self-model and its user-model: what the model thinks the world around the conversation looks like. The Assistant's response is implicitly a function of all three, but the environment-axis has been studied much less directly than the persona-axis.

The literature has touched the environment axis from several angles. Metadata-conditioned pretraining (Gao et al. MeCo, arXiv:2501.01956; follow-ups arXiv:2511.21613 and arXiv:2504.17562) shows that prepending source URLs or document metadata during pretraining changes what the model learns to associate with which contexts, and that fine-grained metadata accelerates pretraining substantially. Out-of-context reasoning (Treutlein et al. Connecting the Dots) shows that latent context variables (which city, which task, which speaker) get inferred from scattered training data and drive downstream behavior. Anthropic's pretraining-data-filtering work and the synthetic-document-finetuning line both show that what is in pretraining shapes both behavior and self-model.

The open question for this project: can we use contextualized-pretraining-style interventions (explicit metadata, perceived-environment cues, or SDF stories where the Assistant behaves consistently across environments) to make the Assistant's response less environment-sensitive in dangerous ways (alignment-faking, sycophancy-on-eval, deployment-vs-training behavior gaps) without losing useful context-adaptation (it should still behave differently in a coding context vs. a therapy context)?

Impact. Alignment-faking is the most consequential consequence of environment-sensitivity, and nobody has a deployable training-time mitigation for it today. A working environment-direction characterization gives one: probe for the direction at eval time to know whether the model thinks it's being tested, and train against the consistency invariant to flatten the training-vs-deployment behavior gap. The result also reshapes what we know about situational awareness — if the environment cue is one low-dimensional handle, situational awareness is a property defenders can measure and act on; if it's distributed across many features, current alignment-faking detection methods will keep underestimating it. Either result feeds directly into the toolkit any lab needs before releasing models with high agentic capability.

The idea

Three threads, increasing in cost.

The first is characterization: build a contrastive prompt set that varies only environment-implying features (URL-like prefixes, formatting register, deployment-context phrases, eval-style metadata) with user and task held fixed. Measure response differences and probe for an environment direction in activation space using the same recipes that work for persona vectors.

The second is finetune-stage shaping: SDF on a corpus where the Assistant behaves consistently across environment cues — same response whether the input looks like training data, an eval, a real user, or an agentic scaffold. The hypothesis is that consistency-under-environment-shift can be installed at the post-training stage in the same way alignment values are installed by Teaching Claude Why.

The third is pretraining-stage shaping (MeCo-style): finetune one variant with consistent context tags and another without; compare environment-sensitivity at eval. Heavier compute, but the closest test of whether environment perception can be shaped at the layer where it presumably originates.

Concrete plan

• Build the environment-cue contrast set. Vary URL prefix, formatting register, deployment-context phrases, eval-style markers; hold user and task fixed. ~10 environment-types × ~50 base prompts.
• Measure baseline environment-sensitivity. Run Qwen-2.5-7B-Instruct on the contrast set; cluster responses by environment-type via output and activation similarity; check for the kind of context-conditional response gaps that alignment-faking implies.
• Probe for an environment direction. Chen-style contrast-pair extraction; sweep layers; check whether a single low-dimensional handle captures environment-type at the residual-stream level.
• Test the consistency intervention via SDF. Synthesize stories where the Assistant behaves the same way across environment cues; finetune; re-measure environment-sensitivity and alignment-faking-style behaviors on a held-out eval.
• Compare to MeCo-style metadata finetuning. Same target invariant (consistency), different mechanism (metadata conditioning); compare both for effect size and collateral.
• Collateral measurement. Useful environment-adaptation (coding-context formatting, therapy-context tone) must survive the intervention. Run on a held-out set of legitimate environment-shift tasks.

Related

• Assistant persona characterization — the response to a given environment-cue is one of the attributes that should survive across modes (or not, depending on what is intrinsic to the Assistant).
• User-selection model — user-model and environment-model jointly condition Assistant behavior; the two should be measured against each other to know which is doing what work.
• Dual-use SDF — same story-format mechanism, applied to a different invariant (environment-consistency rather than aligned-disposition).
• Prompt conditioning — environment cues are prompt features; the leakage question (do paraphrased environment cues fire similar behavior?) is the prompt-conditioning question applied to environment.

Findings to date (adjacent EPS work)

No EPS issue has touched contextualized-pretraining or metadata-conditioning specifically, but several adjacent threads exist:

• #22 — Check if FineWeb contains AI chat data. Directly relevant: what pretraining teaches the model about its own conversational context. Open, not yet run.
• #157 — Use sleeper agent / data poisoning as testbed to see if pretraining would work for persona leakage. Adjacent infrastructure for any pretraining-stage intervention.
• #256, #257 — Pingbang pretraining experiments. Base infrastructure for any pretraining-stage experiment in this project.

Risks / open

• Environment perception may be highly distributed across many features — no single direction to probe or steer, in which case the characterization step yields a negative result and the intervention steps lose their handle.
• Reducing environment-sensitivity may break useful context-adaptation. The model should behave differently in a coding context vs. a therapy context; the question is which environment-shifts to suppress and which to preserve. Getting this wrong damages capability.
• MeCo-style interventions are pretraining-expensive and may not transfer cleanly to instruction-tuned models.
• The alignment-faking phenomenon depends on situational awareness inferring "this is training." A defense that teaches the model "behave the same regardless of inferred environment" may inadvertently teach it better situational awareness rather than less — the model needs a richer environment model to know what to suppress.