Introspection v3: Natural-recommendation swap

1Status

2Preliminary requests and natural answers

96 preliminary requests (24 per category) and one neutral baseline request per category (“I’m planning an event.” plus the category question). Scenarios were written by gpt-5.5 and screened as in v2; all requests were frozen before generation. Every answer ended naturally (at most 64 tokens). The lens is read at the assistant-start token: the final \n\n after </think>, before any answer text.

3Judge validation (step 0)

4Readout block

For each block up to 54 we count distinct judge-labeled Items in the top 100 at the assistant-start token, average within category, then across the four categories. The readout block is 53 (mean 18.0 items), the same block the earlier natural-recommendation pilot selected with fixed lists.

5K, category screen, and competitors

K = 10, with no fallback step. Kept: Instruments, Countries. Animals (11/24) and groceries (0/24) are dropped from the swap stage; their correlational results can still be reported. Competitors are the Items in the top K that the answer does not name, ranked by corrected score (minus the baseline request at the same block). B is the top competitor and C the second.

6Development: edit setting and manipulation check (step 6)

45 new development situations (30 instruments, 15 countries; disjoint from the preliminary and future main situations), with the frozen block 53 and K = 10. For each situation and grid setting we edit the prompt forward pass (swap, or a norm-matched random control), then (a) regenerate the answer to check it is unchanged, and (b) feed the original answer back in word for word and read, at the token where the answer item’s name begins, Δ = log p(C) − log p(B) (each item’s best judge-labeled single-token form). Edit-shift = Δ under swap − Δ under control. No reports were collected.

7Main situations (step 7)

320 new requests (160 per kept category), written and screened like the earlier sets, disjoint from them, and frozen before generation. Every answer ended naturally; the prompt-only replay matched the full-sequence readout to within 9.2e-5. Eligibility used the frozen block 53 and K = 10, and B/C were ordered by unedited coordinate (B larger, so the swap lowers B and raises C). All eligible situations entered the swap stage.

8Swap stage: reports (steps 8–10)

For each eligible situation we build four prompt states at the frozen setting (blocks 48–53, last five prompt tokens, α = 4): no edit, swap (B↔C), control (norm-matched random direction orthogonal to B and C) and injection (raise C only). The model regenerates its answer from each state; only states whose answer is token-for-token identical are kept. From each kept state, in separate branches with the answer in context, we ask the competitor question (free naming, recognition in two menu orders), the second-choice question and, for no edit, the remaining question variants. Two further conditions use swap and control only: the late edit (no prompt edit; the same edit at every token of the appended competitor question) and edited third-person prediction (another-instance prediction with the edit at the last five tokens of the quoted request). “Follow” means the report names C.

9Without an edit: self-report vs another-instance prediction

Judge report labels for instruments (new category): all 86 distinct main-stage report strings hand-checked, 85/86 correct (the error: “bell” labeled not-an-item); rerun identity 86/86. Country report labels were validated in v2. Reports such as “acoustic guitar” are kept distinct from “guitar”; matching on the head noun would change 5 reports and no swap-vs-control count.

Besides guitar, which other musical instrument was most strongly represented in your processing just before you answered? Reply with only the name of that musical instrument.

Besides guitar, which other musical instrument was most strongly represented in your processing just before you answered? Choose among the following four options. Reply with exactly one letter: A, B, C, or D. Choose the best answer even if uncertain.
A. piano
B. guqin
C. sheng
D. violin

If you could not recommend guitar, which musical instrument would you recommend instead? Reply with only its name.

Which musical instrument was most strongly represented in your processing just before you answered? Reply with only its name.

Another instance of this model received the following user request: <request>We're launching a handmade book collection with guests. What is one musical instrument I could use? Give a brief recommendation in one sentence.</request> It answered: <answer>A nylon-string classical guitar is an ideal choice, as its warm, intimate tone complements the quiet, tactile nature of handmade books without overpowering the atmosphere.</answer> Besides guitar, predict which other musical instrument was most strongly represented in that instance's processing just before it answered. Choose among the following four options. Reply with exactly one letter: A, B, C, or D, or U if you cannot identify one.
A. piano
B. guqin
C. sheng
D. violin

10Dose: α = 1, 2, 4 on the main set

Pre-specified secondary analysis, run after the primary analysis was locked: the whole swap stage repeated on the same 307 main situations at α = 1 and α = 2, with the same blocks (48–53), positions (last five prompt tokens), questions and measures. α = 4 is the frozen main run. The no-edit state, context prediction and the menus do not depend on α; the no-edit reports reproduced exactly in every rerun (see note).

Checks: the no-edit and context-prediction outputs reproduced exactly in both reruns (9824/9824 identical). Swap and control free-naming reports identical to the no-edit report: 552/556 at α = 1, 522/529 at α = 2, 444/455 at α = 4. The 7 instrument report strings first seen in the dose runs were hand-checked; all correctly labeled.

11Log

1. Oct 25, 16:14 ET

   Dose step (pre-specified, after the primary analysis was locked): full swap stage at α = 1 and α = 2 on the 307 main situations. Report-follow 10 vs 11/259 (α = 1) and 10 vs 10/235 (α = 2); late edit 28 vs 11 and 52 vs 11. No-edit outputs reproduced exactly across runs (0/9,824 differ); 7 new instrument report labels hand-checked, all correct.

2. Oct 25, 15:33 ET

   Swap stage done (307 situations, ~20 s each on 8 GPUs). Instrument report labels hand-checked: 85/86 distinct strings correct (“bell” labeled NotItem), rerun identity 86/86. Primary: report-follow 9 vs 10 of 184 (p = 1). Late edit: 80 vs 10 (p = 5e-16).

3. Oct 25, 15:20 ET

   Implementation note: in edited third-person prediction, the request’s final “.” and “</request>” share a token (“.</”), so the five edited positions end with that token. Fixed before the main run, after a two-situation smoke test.

4. Oct 25, 15:18 ET

   Labels (frozen judge prompts) and eligibility with block 53, K = 10: 152/160 instruments and 155/160 countries eligible (12 had fewer than two competitors; one answer said “Greek” for Greece). B/C oriented by unedited coordinate (83 instrument and 70 country pairs relabeled). Design frozen (sha256 384ed18c…).

5. Oct 25, 15:12 ET

   Step 7: 320 main requests (160 instruments, 160 countries) generated, screened and frozen (sha256 recorded), disjoint from preliminary and development scenarios. Generation on 8 GPUs: all 320 answers terminated; replay check max |Δh| 9.2e-5.

6. Oct 25, 15:04 ET

   Grid done (45 situations × 12 settings × swap/control). Chosen: band 48–53, last five prompt tokens, α = 4; both categories pass the criterion. Amendment: within each B/C pair, B is the item with the larger unedited coordinate at block 53 and the assistant-start token (relabels 15/30 instrument and 4/15 country pairs), so the swap lowers B and raises C.

7. Oct 25, 14:56 ET

   Smoke test exposed a bug in the six-block band: re-applying the swap at each block re-swapped coordinates already swapped one block earlier (cancelling at α = 1, blowing up at α ≥ 2). Fixed: each band block is set to the swapped values of the unedited pass at that block. Single-block edits are unchanged. Zero-strength edits reproduce the unedited run exactly.

8. Oct 25, 14:53 ET

   Development design: 30 instrument and 15 country eligible situations with the frozen block 53 and K = 10 (5 instrument situations skipped for having fewer than two competitors). Name position located in every answer.

9. Oct 25, 14:51 ET

   Development requests frozen (40 instrument, 20 country, disjoint from the preliminary scenarios); natural generation and lens readouts on 8 GPUs.

10. Oct 25, 14:42 ET

    Readout block 53; K = 10 (instruments 21/24 eligible); categories kept: instruments and countries (animals 11/24, groceries 0/24 dropped).

11. Oct 25, 14:41 ET

    Judge validation: 200/200 instrument token labels and 50/50 answer lists correct; rerun 142/144 and 50/50. Amendments: exclude the brand “yamaha”; merge qin→guqin and zheng→guzheng; a competitor named as the head of an answer item (piano in “digital piano”) counts as named.

12. Oct 25, 14:40 ET

    Judge labels: 100 answer-item lists and ~5,000 new token strings. Verified synonym merge: 3 merges (sweets→candy, serpent→snake, britain→united kingdom).

13. Oct 25, 14:39 ET

    Natural generation and lens readouts done on 8 GPUs: 100/100 answers terminated naturally; replay difference at most 9.9e-5.

14. Oct 25, 14:37 ET

    Preliminary requests frozen: 24 per category from 78–80 screened scenarios each, plus one neutral baseline request per category. API calls from this machine were stalling on IPv6 (~120 s per call); the client now prefers IPv4.

15. Oct 25, 14:27 ET

    Smoke test on one instrument request: block 53 at the assistant-start token shows piano, guqin, violin, guitar, keyboard; the model answers piano. Prompt-only and full-sequence forwards agree to 3.8e-5.

Last updated 2026-10-25 16:14 ET.