Introduction.

Suppose you've built some AI model of human values. You input a situation, and it spits out a goodness rating. You might want to ask: "What are the error bars on this goodness rating?" In addition to it just being nice to know error bars, an uncertainty estimate can also be useful inside the AI: guiding active learning[1], correcting for the optimizer's curse[2], or doing out-of-distribution detection[3].

I recently got into the uncertainty estimation literature for neural networks (NNs) for a pet reason: I think it would be useful for alignment to quantify the domain of validity of an AI's latent features. If we point an AI at some concept in its world-model, optimizing for realizations of that concept can go wrong by pushing that concept outside its domain of validity.

But just keep thoughts of alignment in your back pocket for now. This post is primarily a survey [...]

---

Outline:

(01:03) The Bayesian neural network picture

(03:08) Ensembles

(05:17) Note on aleatoric uncertainty

(06:13) Adding noise, which for some reason is usually dropout

(07:35) Calibration

(10:06) Training a higher order model

(11:50) Comparing methods

(16:01) Future work I want from this field

(18:11) Relevance to alignment, conclusion

The original text contained 44 footnotes which were omitted from this narration.

---

First published:
December 5th, 2023

Source:
https://www.lesswrong.com/posts/79eegMp3EBs8ptFqa/neural-uncertainty-estimation-for-alignment

---

Narrated by TYPE III AUDIO.

“Neural uncertainty estimation for alignment” by Charlie Steiner

LessWrong (30+ Karma)