Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

TL;DR: We use SGD to find sparse connections between features; additionally a large fraction of features between the residual stream & MLP can be modeled as linearly computed despite the non-linearity in the MLP. See linear feature section for examples.

Special thanks to fellow AISST member, Adam Kaufman, who originally thought of the idea of learning sparse connections between features & to Jannik Brinkmann for training these SAE's.

Sparse AutoEncoders (SAE)'s are able to turn the activations of an LLM into interpretable features. To define circuits, we would like to find how these features connect to each other. SAE's allowed us to scalably find interpretable features using SGD, so why not use SGD to find the connections too?

A layer in Pythia-70M, with SAE's before & after the MLP. Note, this is different than the typical Transformer Architecture [...]

---

Outline:

(03:18) What percentage of Features are Linear?

(04:27) Adding in Sparsity

(05:31) Sparse Linear Feature Connections

(05:52) OR Example

(07:16) Negative Weight Example

(09:11) Sparse MLP Feature Connections

(13:15) SAEs on SAEs

(15:33) Please Help

(17:21) The Grand Plan

(19:12) Code

(19:58) Appendices

(20:05) Failed MLP sweep

(20:22) Attention

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
December 9th, 2023

Source:
https://www.lesswrong.com/posts/7fxusXdkMNmAhkAfc/finding-sparse-linear-connections-between-features-in-llms

---

Narrated by TYPE III AUDIO.

“Finding Sparse Linear Connections between Features in LLMs” by Logan Riggs, Sam Mitchell, Eccentricity

LessWrong (30+ Karma)