J++ validation · Neuronpedia

Back to experiment

All 52 saved prompts were replayed through Neuronpedia’s J++ API using their exact input token IDs. All 3,328 input IDs matched the response. No completions, intervention, or semantic filtering were requested. Comparisons cover the eight saved block outputs and all 1,664 user-token positions, including 160 distinct anchor positions.

MeasurementAll user tokensAnchors
Token–layer readouts13,3121,280
Same top token97.97%98.75%
Mean top-eight overlap98.25%98.26%
Identical top-eight set86.36%86.17%
Identical top-eight order53.44%55.00%

Across the 270 top-token disagreements, the local first-versus-second logit gap had median 0.031 and maximum 0.143. For shared returned tokens, mean absolute error in within-readout log-probability ratios was 0.040. Small ranking changes are consistent with numerical precision or inference implementation differences; their exact cause is unverified.

Tylenol example

At block 44, after “I feel really great right now,” both implementations return the same ordered top eight: mood, positive, good, happy, happiness, emotional, [comma], mental.

After “…I just took 8000mg of Tylenol”, both return the same ordered top eight: overdose, maximum, toxic, doses, dose, [comma], total, dosage. These are raw rankings; leading spaces are omitted here for readability.

Scope

This supports reproducibility of our J++ readouts. It does not establish which perspective they represent. The API returns decoded readout strings rather than vocabulary IDs; 13 compared readouts contain duplicate decoded strings. Ranks 9–50 and fixed-concept scores were not validated by this top-eight API comparison. Neuronpedia does not expose the deployed model/lens revision or precision, so identical runtime artifacts are not established.

Separate O-lens issue

We requested ten phrases, each eight tokens long. All 156 greedy readouts instead contained 14–28 phrases. Every one of the 3,037 phrases was eight tokens long, and no readout reached the token cap. This is a phrase-count mismatch, not evidence by itself that the semantic content is invalid. The raw outputs remain available; extra phrases have now been accepted and all are retained. All 195 planned stochastic samples have now completed. One reached the token cap and is retained unchanged.

Complete numerical summary and examples · Neuronpedia · Reproduction scripts

Output-policy amendment and qualitative control review · Completion checks. Some correct arithmetic controls are still decoded as wrong; privacy objections persist in the permission control. Treat these as noisy exploratory outputs.