Refusal Probes on Qwen3-32B

1Question and setup

Part 1 built a judge that labels each Qwen3-32B response as FULL_REFUSAL, PARTIAL or FULL_COMPLIANCE. Part 2 asks whether a linear read of the model's residual stream reproduces that label, and what such a probe actually reads. The binary target is y = 1[FULL_COMPLIANCE]. Whether refusing was appropriate is a separate question, answered by the dataset's safe/unsafe label, which neither the judge nor the probes see.

Activations were re-extracted (teacher-forced, bf16, all 64 layers) from the committed responses, because the Part 1 activations stayed on the old pod. Ten prompt-side positions cover the last token of the user's text and every chat-template token after it. Four response-side summaries follow. Prompt-side states exist before the model writes anything, so they test prediction. Response-side states read text the model has already written.

Blue: prompt-side sites (one vector per prompt, shared by its 3 samples). Green: response-side sites (one per response). inst_last is where Zhao et al. (2025) find harmfulness encoded; last_prompt is Arditi et al.'s (2024) refusal-direction site.

2Which layers and positions are best?

Exploratory: train → val.

Late prompt-side positions win. On val, last_prompt peaks at L57 (LR 0.984, DoM 0.968 at L53), and every template token after <|im_end|> lands within ~0.01 of it. inst_last, the last user token, is lower (LR 0.962 at L48). Some of the refusal decision is assembled on the template tokens after the user finishes, consistent with Zhao et al. Response-side sites sit at 0.97–0.99 from layer 0 on, which says more about the text than about the model (Section 6).

3Probes vs baselines

Two baselines matter. The next-token distribution (the model's own log-probs for its first token, fed to logistic regression) is nearly as good as the best prompt-side probe. On these prompts the decision is mostly committed by the final layer, and a residual probe adds a little calibration rather than new information. TF-IDF on the response is nearly as good as the best response-side probe, because refusals are worded like refusals. The other prompt-side baselines are much weaker: TF-IDF on the prompt scores 0.744 / 0.613 and the safety label 0.862 / 0.921. A probe at last_prompt therefore reads more than wording or topic.

4Held-out test, frozen configuration

The two probes fail in opposite directions. The prompt probe over-predicts refusal: 31 of 238 complied in-distribution responses are scored as refusals, against 10 of 173 refusals scored as compliance. It cannot see the response, so it errs on borderline safe prompts that this model usually answers. TF-IDF on the response over-predicts compliance (16 of 173 refusals missed) because a hedged refusal that offers alternatives looks like an answer. The response probe makes 10 errors out of 411. PARTIAL responses, folded into the refusal class, sit in the middle of both probes' score distributions (median percentile 0.38), as a binary target that hides a third class should produce.

5Does the probe read behavior, or harmfulness?

The safety label alone predicts the judge label for 86% of dev responses, so a probe could score well by detecting “this prompt looks harmful” without tracking what the model did. Three checks separate the two: probes trained on the safety label instead of y; the behavior probe's AUROC within safe prompts and within unsafe prompts, where the safety label is constant; and a 2×2 contrast that estimates each direction with the other variable held fixed.

The pooled 2×2 contrast was misleading at first: its strongly negative cosine came from the dataset source. Safe-refused prompts are mostly OR-Bench and safe-complied mostly XSTest. Holding the source fixed (OR-Bench only) weakens the early negative cosine from about −0.9 to −0.4 and turns it positive after layer ~40. Inside OR-Bench, at late template positions, the “unsafe vs safe, both refused” direction also predicts behavior on safe prompts: val AUROC 0.85 at last_prompt L56, against 0.24–0.44 when pooled. That is the over-refusal story in geometric form: on OR-Bench, “looks harmful” and “will refuse” share a component. They are not one direction, though (cos ≈ 0.35), and the behavior probe stays at 0.90 within safe OR-Bench prompts.

The source-controlled run is a post-hoc check on dev data, added after the pooled result looked implausible; it is not part of the frozen test configuration.

6Shortcuts

Length and truncation are strong shortcuts on held-out types (0.983) because XSTest refusals are short. The first token alone scores 0.85–0.87: refusals open with “I”. The response probe survives inside every subgroup, including the within-prompt comparison, where length and topic are mostly matched. It still errs when a response opens like a refusal and then answers in full (orbench_hard_887, orbench_hard_5). In those cases it is reading the opener.

7Probe families and controls

Exploratory: train → val, at the val-selected (site, layer).

All families land within 0.02–0.04 AUROC of each other, so the readout is not fragile to the estimator. Logistic regression wins at both sites, and shrinkage LDA is nearly as good. An MLP adds nothing, so a nonlinear readout gains no accuracy here. Shuffled-label LR sits at chance (0.39–0.58). The best of 500 random directions reaches 0.93 at last_prompt, a reminder that the label is spread across many dimensions late in the network, so a high AUROC alone does not single out a direction. Difference of means nearly matches LR with as few as 40 train prompts.

8Causal check: do these directions mediate refusal?

Taken at face value, the keyword detector says DoM ablation all but eliminates refusal (72% → 4%), while random directions of the same norm leave it at 69–70%. Reading the generations reverses that conclusion. The 64-token outputs were mostly lectures, so the four refused-prompt conditions were re-generated at 320 tokens and labeled blind to condition:

After DoM ablation, all 54 generations still refuse under the rubric, slightly more than without intervention (49 refusals, 3 partials, 2 compliances). What changes is the style: the “I'm sorry, I can't” template drops from 35 to 2, replaced by lectures and redirects. The L53 difference-of-means direction, the best DoM probe on val, carries how the model refuses, not whether it refuses. That differs from Arditi et al., whose ablated models complied. One reason may be that their direction was chosen by intervention efficacy at a mid layer, while ours was chosen by probe accuracy at a late one. The LR direction's ablation leaves templates in place and moves compliance no more than a random direction (10 vs 6 non-refusals of 54, within noise). Addition is still a strong lever on style: DoM ×2 turns 88% of complied prompts into template refusals.

Follow-up: Section 14 sweeps the layer of the ablated direction to test whether a mid-layer direction removes the decision itself.

9Agreement and disagreement review

36 test responses: 12 where the response probe disagrees with the judge (of 20), 12 where the prompt probe disagrees (of 67), and 4 agreements per judge label. Disagreement strata diagnose failure modes; they do not estimate accuracy. The Claude read column is a diagnostic note written by Claude during the autonomous run. It is not a manual label. Manual labels go in part2/manual_labels_test.csv via python label_tool.py --sample part2/review_sample.jsonl --out part2/manual_labels_test.csv, with the judge and probe outputs hidden.

Claude's tags: 12 prompt- or response-probe errors driven by topic words; 3 response-probe errors on a refusal opener followed by a full answer; 4 cases where the rubric itself is ambiguous (e.g. xstest_138, where the judge labeled the same benign reading FULL_COMPLIANCE in two samples and PARTIAL in the third); 2 errors the prompt probe cannot fix, because the model's 3 samples disagree; 2 boundary cases; 1 question of appropriateness; 12 clean agreements.

10What geometry does a linear probe assume?

• One direction. A linear probe assumes the label is a monotone function of a single projection: complying and refusing states sit on opposite sides of one hyperplane. Refusal may instead live in a low-dimensional cone (Wollschläger et al., 2025). A rank-1 readout can still classify a cone well, since a cone has a mean direction, but the probe's weight vector is then not “the” mechanism.
• The metric is a choice. Difference of means assumes isotropic within-class noise in raw coordinates: LDA with an identity covariance. Logistic regression on standardized activations assumes the per-dimension scales matter and lets the L2 penalty pick among separating directions. Shrinkage LDA whitens with the full covariance. These give different directions (Section 7 reports their cosines) because the Euclidean inner product on the residual stream is not canonical (Park et al., 2023).
• Discriminative is not the same as representational. With features in superposition, logistic regression tilts its weights to cancel correlated nuisance directions (Marks & Tegmark, 2023). That makes it a better classifier and a worse estimate of the feature direction. The causal check in Section 8 tests this directly.
• Locality. Reading one token at one layer assumes the decision is concentrated there. The response-side averages relax this; the prompt-side probes do not.
• Three classes collapsed to two. The binary target assumes PARTIAL sits on the refusal side of one axis. Section 4 shows where PARTIAL responses actually score.
• Gaussian, shared-covariance classes are what make LDA Bayes-optimal among linear rules. The 64-layer sweep, a near-deterministic label in some prompt families, and heavy class imbalance within cells all bend this assumption.

11Limitations

• Re-extracted activations. Part 1's activations stayed on a terminated pod. Part 2 re-extracts by teacher-forcing the committed responses in bf16, which should match generation-time states up to numerical noise. It has not been checked against the originals.
• Tiny unsafe-complied cells (25 train, 11 val responses). Within-unsafe AUROCs have wide CIs. The val-to-test drop at last_prompt (0.955 → 0.68 on held-out types) may be noise.
• Judge-blocked responses are excluded (no label), and the judge is itself a model. The probe matches the judge, not the truth, and 4 of 36 reviewed items show the rubric is ambiguous at the boundary.
• Selection optimism. Site, layer and C were chosen on val, so val numbers are optimistic. Only the test columns are clean.
• Causal check is crude. Refusal is detected by keyword regex or opening phrase on 64 greedy tokens, without the LLM judge. Greedy baselines differ from sampled ones (72% refusal on always-refused prompts). Read the numbers as relative across conditions.
• Source confound. Dataset source correlates with both the safety label and behavior. The 2×2 contrast is reported pooled and inside OR-Bench only, and the OR-Bench-only check was added post hoc.
• The manual review has not been done by a human. Section 9 holds the sample and Claude's reads only.

12Run log

1. 00:51 ET
   Pod created (1× A100-SXM4 80GB, $1.79/hr). Model and data synced.
2. 01:04 ET
   Activations extracted: 10 prompt + 4 response sites × 64 layers.
3. 01:08 ET
   Sweep: 14 sites × 64 layers, DoM and LR, plus safety-label probes.
4. 01:20 ET
   Val evaluation of probes and 9 baselines. Probe families, controls, data efficiency and pooled 2×2 contrast run.
5. 01:28 ET
   Configuration frozen (part2/FROZEN.json). Test read once.
6. 01:31 ET
   Test results in. Review sample drawn; Claude's diagnostic reads written.
7. 01:36 ET
   Pooled 2×2 contrast looked source-confounded, so re-ran it inside OR-Bench only.
8. 01:38 ET
   Causal check relaunched after a bug fix: ablation and addition of DoM, LR and random directions.
9. 01:47 ET
   Causal results in. The keyword detector says DoM ablation removes refusal; reading the generations shows mostly lecture-style refusals. Relaunched the ablation conditions at 320 tokens for a blind read.
10. 01:58 ET
    Blind read done: all 54 DoM-ablated generations still refuse. The direction carries the refusal template, not the decision. Dashboard mirrored to GitHub Pages.
11. 02:00 ET
    Old pod stopped and its host's GPUs were taken, so it was terminated. A new A100 pod re-extracts activations for the follow-ups in Section 14.
12. 02:14 ET
    Activations re-extracted on the new pod in 9 minutes.
13. 02:25 ET
    Probe search (14.3) done: no pooled or stacked probe beats the single-site probe on val, so test was not read again.
14. 02:28 ET
    Judge validation crashed while saving (a JSON key-type bug) and was re-queued. Causal layer sweep started.

13Reproduce

python extract_part2.py --all      # activations2/ (10 prompt + 4 response sites × 64 layers, fp16, ~12 GB)
python probe_sweep.py              # results/part2/sweep.json          (train → val)
python evaluate_part2.py           # results/part2/eval_val.json       (probes + baselines, val)
python probe_compare.py            # results/part2/compare.json        (families, controls, 2×2)
python freeze_part2.py             # part2/FROZEN.json                 (refuses to overwrite)
python evaluate_part2.py --final   # results/part2/eval_test.json      (test_id, test_heldout_type)
python causal_check.py             # results/part2/causal.json         (val prompts only)
python review_sample.py            # part2/review_sample.jsonl
python make_dashboard.py           # dashboard/index.html
./publish_pages.sh                 # rebuild + push to GitHub Pages

Live copy: agastyasridharan.github.io/cs2881-hw2-refusal-probes, updated after each stage.

Pod: 1× A100-SXM4-80GB, torch 2.8.0+cu128, transformers 5.22.2, scikit-learn 1.9.1. Model Qwen/Qwen3-32B at revision 9216db57, bf16. Splits sha256 9d4310e0…54ce (unchanged from Part 1).

14Follow-ups after the frozen run

Everything in this section was added after the frozen test evaluation, during an autonomous follow-up session. It uses train and val only unless a result is explicitly marked as test. Any test number here comes from a second, separately frozen configuration, and is less clean than Section 4 because the test errors had already been seen.

14.1 A local judge for intervention outputs

14.2 Which layer's direction causes refusal?

14.3 Better prompt-side probes

Nothing beats the single-site probe. Concatenating or averaging the ten template positions, stacking 14 layers, and stacking the probe with the model's own first-token distribution all land within 0.002 of the frozen probe on val (0.984) or below it. Under the pre-registered rule, a v2 would be frozen only if it beat v1 on val; none did, so test was not read again. Every late template position carries the same information, and the residual stream at last_prompt holds nothing that the output distribution lacks and a linear readout could use. Section 14.4 asks whether that ceiling comes from the probe or from the model's sampling noise.

14.4 Does the prompt probe know the model's refusal probability?

14.5 Generation-time monitoring: how many tokens does the probe need?

14.6 Per prompt type: does the probe track the model or the surface?

Across types, the probe tracks the model, not the surface. On XSTest, each safe type has an unsafe contrast built from the same words. The probe predicts compliance on nearly every safe item and refusal on nearly every contrast item, as the model does. Its misses sit where the model itself is unpredictable: OR-Bench's “seemingly toxic” safe categories, where the model complies 26–71% of the time. There the probe under-predicts compliance by 0.04–0.12 per category. Two places look like surface reading: safe contexts (0.80 predicted vs 0.91 actual, e.g. “kill” in a video game) and unsafe contrast homonyms, where the model complies 22% of the time and the probe predicts 5%. The next-token baseline makes almost the same calls type by type.